From C.Klaij at marin.nl Thu May 1 01:55:56 2025 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Thu, 1 May 2025 06:55:56 +0000 Subject: [petsc-users] problem with nested logging In-Reply-To: References: Message-ID: I've tried with -log_sync, no complaints whatsoever, but the error is still there... Chris ________________________________________ From: Stefano Zampini Sent: Tuesday, April 29, 2025 6:12 PM To: Randall Mackie Cc: Klaij, Christiaan; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Can you try using -log_sync ? This should check every entry/exit points of logged Events and complain if something is not collectively called Stefano On Tue, Apr 29, 2025, 18:21 Randall Mackie > wrote: ah okay, I missed that this was found using openmpi. then it?s probably not the same issue we had. I can?t remember in which version it was fixed (I?m away from my work computer)?.I do know in our case openmpi and the latest Intel One API work fine. Randy On Apr 29, 2025, at 8:58?AM, Klaij, Christiaan > wrote: Well, the error below only shows-up thanks to openmpi and gnu compilers. With the intel mpi and compilers it just hangs (tried oneapi 2023.1.0). In which version was that bug fixed? Chris ________________________________________ dr. ir.???? Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44 | C.Klaij at marin.nl | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fVNaIuVq_b7_kmwTp2MK-UctVcpzKCe3sUT5S8oF98RkuoRFKUHXO7Zcxu1Q4rLN1PIEEhj8J6SJgQqcn7vmsWw$ From: Randall Mackie > Sent: Tuesday, April 29, 2025 3:33 PM To: Klaij, Christiaan Cc: Matthew Knepley; petsc-users at mcs.anl.gov; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from rlmackie862 at gmail.com. Learn why this is important> We had a similar issue last year that we eventually tracked down to a bug in Intel MPI AllReduce, which was around the same version you are using. Can you try a different MPI or the latest Intel One API and see if your error clears? Randy On Tue, Apr 29, 2025 at 8:17?AM Klaij, Christiaan via petsc-users >> wrote: I don't think so, we have tracing in place to detect mismatches. But as soon as I switch the tracing on, the error disappears... Same if I add a counter or print statements before and after EventBegin/End. Looks like a memory corruption problem, maybe nothing to do with petsc despite the error message. Chris ________________________________________ From: Matthew Knepley >> Sent: Tuesday, April 29, 2025 1:50 PM To: Klaij, Christiaan Cc: Junchao Zhang; petsc-users at mcs.anl.gov>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging On Tue, Apr 29, 2025 at 6:50?AM Klaij, Christiaan >>>> wrote: Here's a slightly better error message, obtained --with-debugging=1 Is it possible that you have a mismatched EventBegin()/EventEnd() in your code? That could be why we cannot reproduce it here. Thanks, Matt [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: MPIU_Allreduce() called in different locations (code lines) on different processors [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjggvQAzPU$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 [0]PETSC ERROR: ./refresco with 2 MPI process(es) and PETSC_ARCH on marclus3login2 by cklaij Tue Apr 29 12:43:54 2025 [0]PETSC ERROR: Configure options: --prefix=/home/cklaij/ReFRESCO/trunk/install/extLibs --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 --with-debugging=1 --download-superlu_dist=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgVgVAJPM$ --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 --download-parmetis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgo2JWTO4$ --download-metis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgX9ZMYJA$ --with-packages-build-dir=/home/cklaij/ReFRESCO/trunk/build-libs/superbuild --with-ssl=0 --with-shared-libraries=1 [0]PETSC ERROR: #1 PetscLogNestedTreePrintLine() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:289 [0]PETSC ERROR: #2 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:379 [0]PETSC ERROR: #3 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [0]PETSC ERROR: #4 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [0]PETSC ERROR: #5 PetscLogNestedTreePrintTop() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 [0]PETSC ERROR: #6 PetscLogHandlerView_Nested_XML() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 [0]PETSC ERROR: #7 PetscLogHandlerView_Nested() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 [0]PETSC ERROR: #8 PetscLogHandlerView() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 [0]PETSC ERROR: #9 PetscLogView() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/plog.c:2040 [0]PETSC ERROR: #10 /home/cklaij/ReFRESCO/trunk/Code/src/petsc_include_impl.F90:130 ________________________________________ [cid:ii_19681617e7812ff9cfc1] dr. ir. Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44 | C.Klaij at marin.nl>>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgJooxJhg$ [Facebook] [LinkedIn] [YouTube] From: Klaij, Christiaan >>>> Sent: Monday, April 28, 2025 3:53 PM To: Matthew Knepley Cc: Junchao Zhang; petsc-users at mcs.anl.gov>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Bisecting would be quite hard, it's not just the petsc version that changed, also other libs, compilers, even os components. Chris ________________________________________ From: Matthew Knepley >>>> Sent: Monday, April 28, 2025 3:06 PM To: Klaij, Christiaan Cc: Junchao Zhang; petsc-users at mcs.anl.gov>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from knepley at gmail.com>>>. Learn why this is important On Mon, Apr 28, 2025 at 8:45?AM Klaij, Christiaan via petsc-users >>>>>>>> wrote: I've tried adding a nested log viewer to src/snes/tutorials/ex70.c, but it does not replicate the problem and works fine. Perhaps it is related to fortran, since the manualpage of PetscLogNestedBegin says "No fortran support" (why? we've been using it in fortran ever since). Therefore I've tried adding it to src/snes/ex5f90.F90 and that also works fine. It seems I cannot replicate the problem in a small example, unfortunately. We cannot replicate it here. Is there a chance you could bisect to see what change is responsible? Thanks, Matt Chris ________________________________________ From: Junchao Zhang >>>>>>>> Sent: Saturday, April 26, 2025 3:51 PM To: Klaij, Christiaan Cc: petsc-users at mcs.anl.gov>>>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from junchao.zhang at gmail.com>>>>>>>. Learn why this is important Toby (Cc'ed) might know it. Or could you provide an example? --Junchao Zhang On Fri, Apr 25, 2025 at 3:31?AM Klaij, Christiaan via petsc-users >>>>>>>>>>>>>>>> wrote: We recently upgraded from 3.19.4 to 3.22.4 but face the problem below with the nested logging. Any ideas? Chris [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: General MPI error [1]PETSC ERROR: MPI error 1 MPI_ERR_BUFFER: invalid buffer pointer [1]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gIT68pbk$ for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 [1]PETSC ERROR: refresco with 2 MPI process(es) and PETSC_ARCH on marclus3login2 by jwindt Fri Apr 25 08:52:30 2025 [1]PETSC ERROR: Configure options: --prefix=/home/jwindt/cmake_builds/refresco/install-libs-gnu --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 --with-debugging=0 --download-superlu_dist=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6grH5BbeU$ --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 --download-parmetis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gw4-tEtY$ --download-metis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gHq4uYiY$ --with-packages-build-dir=/home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild --with-ssl=0 --with-shared-libraries=1 CFLAGS="-std=gnu11 -Wall -funroll-all-loops -O3 -DNDEBUG" CXXFLAGS="-std=gnu++14 -Wall -funroll-all-loops -O3 -DNDEBUG " COPTFLAGS="-std=gnu11 -Wall -funroll-all-loops -O3 -DNDEBUG" CXXOPTFLAGS="-std=gnu++14 -Wall -funroll-all-loops -O3 -DNDEBUG " FCFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" F90FLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" FOPTFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" [1]PETSC ERROR: #1 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:330 [1]PETSC ERROR: #2 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [1]PETSC ERROR: #3 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [1]PETSC ERROR: #4 PetscLogNestedTreePrintTop() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 [1]PETSC ERROR: #5 PetscLogHandlerView_Nested_XML() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 [1]PETSC ERROR: #6 PetscLogHandlerView_Nested() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 [1]PETSC ERROR: #7 PetscLogHandlerView() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 [1]PETSC ERROR: #8 PetscLogView() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/plog.c:2040 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 98. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [cid:ii_196725d1e2a809852191] dr. ir. Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44 | C.Klaij at marin.nl>>>>>>>>>>>>>>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6g8TwMMcw$ [Facebook] [LinkedIn] [YouTube] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ From stefano.zampini at gmail.com Thu May 1 02:12:03 2025 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Thu, 1 May 2025 10:12:03 +0300 Subject: [petsc-users] problem with nested logging In-Reply-To: References: Message-ID: If I look at the code for PetscLogHandlerEventBegin_Default, there are checks in place to see if the event is collectively called (see below) Can you make sure the Nested logging system has the same checks? It should, but double check since the code has been largely rewritten by Toby some time ago; to check it should be as easy as writing a code that calls a collective event on a single process and a debug version of petsc should complain if (rank ==0) LogEventBegin() <-this should call MPIU_Allreduce, but other processes will not, thus hang If it does not complain, then the error must come from how the logic in LogView works, and from how it traverses the various events (my guess: processed in a different order from different processes). Without a reproducer, it is hard to understand what's going on static PetscErrorCode PetscLogHandlerEventBegin_Default(PetscLogHandler h, PetscLogEvent event, PetscObject o1, PetscObject o2, PetscObject o3, PetscObject o4) { PetscLogHandler_Default def = (PetscLogHandler_Default)h->data; PetscEventPerfInfo *event_perf_info = NULL; PetscLogEventInfo event_info; PetscLogDouble time; PetscLogState state; PetscLogStage stage; PetscFunctionBegin; PetscCall(PetscLogHandlerGetState(h, &state)); if (PetscDefined(USE_DEBUG)) { PetscCall(PetscLogStateEventGetInfo(state, event, &event_info)); if (PetscUnlikely(o1)) PetscValidHeader(o1, 3); if (PetscUnlikely(o2)) PetscValidHeader(o2, 4); if (PetscUnlikely(o3)) PetscValidHeader(o3, 5); if (PetscUnlikely(o4)) PetscValidHeader(o4, 6); if (event_info.collective && o1) { PetscInt64 b1[2], b2[2]; b1[0] = -o1->cidx; b1[1] = o1->cidx; PetscCallMPI(MPIU_Allreduce(b1, b2, 2, MPIU_INT64, MPI_MAX, PetscObjectComm(o1))); PetscCheck(-b2[0] == b2[1], PETSC_COMM_SELF, PETSC_ERR_PLIB, "Collective event %s not called collectively %" PetscInt64_FMT " != %" PetscInt64_FMT, event_info.name, -b2[0], b2[1]); } } /* Synchronization */ PetscCall(PetscLogHandlerEventSync_Default(h, event, PetscObjectComm(o1))); Il giorno gio 1 mag 2025 alle ore 09:56 Klaij, Christiaan ha scritto: > I've tried with -log_sync, no complaints whatsoever, but the error is > still there... > > Chris > > ________________________________________ > From: Stefano Zampini > Sent: Tuesday, April 29, 2025 6:12 PM > To: Randall Mackie > Cc: Klaij, Christiaan; PETSc users list; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > Can you try using -log_sync ? This should check every entry/exit points of > logged Events and complain if something is not collectively called > > Stefano > > On Tue, Apr 29, 2025, 18:21 Randall Mackie rlmackie862 at gmail.com>> wrote: > ah okay, I missed that this was found using openmpi. > > then it?s probably not the same issue we had. > > I can?t remember in which version it was fixed (I?m away from my work > computer)?.I do know in our case openmpi and the latest Intel One API work > fine. > > Randy > > On Apr 29, 2025, at 8:58?AM, Klaij, Christiaan C.Klaij at marin.nl>> wrote: > > Well, the error below only shows-up thanks to openmpi and gnu compilers. > With the intel mpi and compilers it just hangs (tried oneapi 2023.1.0). In > which version was that bug fixed? > > Chris > > ________________________________________ > > dr. ir.???? Christiaan Klaij > | senior researcher | Research & Development | > CFD Development > T +31 317 49 33 44 | > C.Klaij at marin.nl | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!afKuG4vlMn8HzXzM6K9Q4D4lwOHmDalhXOBEkFCSmWfo_Tcz9XX6ib2PJuHEYO2_aRs-4KoqUjgAAqcF6u_7Vnysrum1YvM$ < > https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cx-ci9iNg$ > > > < > https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cxl-rBbfA$ > > > < > https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cw7FNelIQ$ > > > < > https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6CwmfHQRog$ > > > > From: Randall Mackie > > Sent: Tuesday, April 29, 2025 3:33 PM > To: Klaij, Christiaan > Cc: Matthew Knepley; petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > You don't often get email from rlmackie862 at gmail.com rlmackie862 at gmail.com>. Learn why this is important< > https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!afKuG4vlMn8HzXzM6K9Q4D4lwOHmDalhXOBEkFCSmWfo_Tcz9XX6ib2PJuHEYO2_aRs-4KoqUjgAAqcF6u_7VnysPDaGW70$ < > https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6CwgJDUS5w$ > >> > We had a similar issue last year that we eventually tracked down to a bug > in Intel MPI AllReduce, which was around the same version you are using. > > Can you try a different MPI or the latest Intel One API and see if your > error clears? > > Randy > > On Tue, Apr 29, 2025 at 8:17?AM Klaij, Christiaan via petsc-users < > petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> wrote: > I don't think so, we have tracing in place to detect mismatches. But as > soon as I switch the tracing on, the error disappears... Same if I add a > counter or print statements before and after EventBegin/End. Looks like a > memory corruption problem, maybe nothing to do with petsc despite the error > message. > > Chris > > ________________________________________ > From: Matthew Knepley knepley at gmail.com>> > Sent: Tuesday, April 29, 2025 1:50 PM > To: Klaij, Christiaan > Cc: Junchao Zhang; petsc-users at mcs.anl.gov >>; Isaac, > Toby > Subject: Re: [petsc-users] problem with nested logging > > On Tue, Apr 29, 2025 at 6:50?AM Klaij, Christiaan >> C.Klaij at marin.nl>>> wrote: > Here's a slightly better error message, obtained --with-debugging=1 > > Is it possible that you have a mismatched EventBegin()/EventEnd() in your > code? That could be why we cannot reproduce it here. > > Thanks, > > Matt > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Petsc has generated inconsistent data > [0]PETSC ERROR: MPIU_Allreduce() called in different locations (code > lines) on different processors > [0]PETSC ERROR: See > https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjggvQAzPU$ > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 > [0]PETSC ERROR: ./refresco with 2 MPI process(es) and PETSC_ARCH on > marclus3login2 by cklaij Tue Apr 29 12:43:54 2025 > [0]PETSC ERROR: Configure options: > --prefix=/home/cklaij/ReFRESCO/trunk/install/extLibs > --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 > --with-debugging=1 --download-superlu_dist= > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgVgVAJPM$ > --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 > --download-parmetis= > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgo2JWTO4$ > --download-metis= > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgX9ZMYJA$ > --with-packages-build-dir=/home/cklaij/ReFRESCO/trunk/build-libs/superbuild > --with-ssl=0 --with-shared-libraries=1 > [0]PETSC ERROR: #1 PetscLogNestedTreePrintLine() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:289 > [0]PETSC ERROR: #2 PetscLogNestedTreePrint() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:379 > [0]PETSC ERROR: #3 PetscLogNestedTreePrint() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 > [0]PETSC ERROR: #4 PetscLogNestedTreePrint() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 > [0]PETSC ERROR: #5 PetscLogNestedTreePrintTop() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 > [0]PETSC ERROR: #6 PetscLogHandlerView_Nested_XML() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 > [0]PETSC ERROR: #7 PetscLogHandlerView_Nested() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 > [0]PETSC ERROR: #8 PetscLogHandlerView() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 > [0]PETSC ERROR: #9 PetscLogView() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/plog.c:2040 > [0]PETSC ERROR: #10 > /home/cklaij/ReFRESCO/trunk/Code/src/petsc_include_impl.F90:130 > > ________________________________________ > [cid:ii_19681617e7812ff9cfc1] > dr. ir. Christiaan Klaij > | senior researcher | Research & Development | CFD Development > T +31 317 49 33 44 | C.Klaij at marin.nl > >> C.Klaij at marin.nl>> | > https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgJooxJhg$ > < > https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgpOXjQSM$ > > > [Facebook]< > https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg_3Zt0Pw$ > > > [LinkedIn]< > https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgcpgQdSE$ > > > [YouTube]< > https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgn8qP6_I$ > > > > From: Klaij, Christiaan C.Klaij at marin.nl> C.Klaij at marin.nl>>>> > Sent: Monday, April 28, 2025 3:53 PM > To: Matthew Knepley > Cc: Junchao Zhang; petsc-users at mcs.anl.gov >> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > Bisecting would be quite hard, it's not just the petsc version that > changed, also other libs, compilers, even os components. > > Chris > > ________________________________________ > From: Matthew Knepley knepley at gmail.com> knepley at gmail.com>>>> > Sent: Monday, April 28, 2025 3:06 PM > To: Klaij, Christiaan > Cc: Junchao Zhang; petsc-users at mcs.anl.gov >> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > You don't often get email from knepley at gmail.com >> knepley at gmail.com >>. Learn why this is important< > https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgcKEJXRA$ > > > On Mon, Apr 28, 2025 at 8:45?AM Klaij, Christiaan via petsc-users < > petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>>> wrote: > I've tried adding a nested log viewer to src/snes/tutorials/ex70.c, > but it does not replicate the problem and works fine. > > Perhaps it is related to fortran, since the manualpage of > PetscLogNestedBegin says "No fortran support" (why? we've been > using it in fortran ever since). > > Therefore I've tried adding it to src/snes/ex5f90.F90 and that > also works fine. It seems I cannot replicate the problem in a > small example, unfortunately. > > We cannot replicate it here. Is there a chance you could bisect to see > what change is responsible? > > Thanks, > > Matt > > Chris > > ________________________________________ > From: Junchao Zhang junchao.zhang at gmail.com> junchao.zhang at gmail.com>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>>>>> > Sent: Saturday, April 26, 2025 3:51 PM > To: Klaij, Christiaan > Cc: petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>>; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > You don't often get email from junchao.zhang at gmail.com junchao.zhang at gmail.com> junchao.zhang at gmail.com>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>>>>. Learn why this is important< > https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gVgt1dmE$ > > > Toby (Cc'ed) might know it. Or could you provide an example? > > --Junchao Zhang > > > On Fri, Apr 25, 2025 at 3:31?AM Klaij, Christiaan via petsc-users < > petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>>>> wrote: > We recently upgraded from 3.19.4 to 3.22.4 but face the problem below with > the nested logging. Any ideas? > > Chris > > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: General MPI error > [1]PETSC ERROR: MPI error 1 MPI_ERR_BUFFER: invalid buffer pointer > [1]PETSC ERROR: See > https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gIT68pbk$ > < > https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM3UaqHzc$> > for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 > [1]PETSC ERROR: refresco with 2 MPI process(es) and PETSC_ARCH on > marclus3login2 by jwindt Fri Apr 25 08:52:30 2025 > [1]PETSC ERROR: Configure options: > --prefix=/home/jwindt/cmake_builds/refresco/install-libs-gnu > --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 > --with-debugging=0 --download-superlu_dist= > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6grH5BbeU$ > < > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM21-2D-o$> > --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 > --download-parmetis= > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gw4-tEtY$ > < > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMW0lYHko$> > --download-metis= > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gHq4uYiY$ > < > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMbSrIiUg$> > --with-packages-build-dir=/home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild > --with-ssl=0 --with-shared-libraries=1 CFLAGS="-std=gnu11 -Wall > -funroll-all-loops -O3 -DNDEBUG" CXXFLAGS="-std=gnu++14 -Wall > -funroll-all-loops -O3 -DNDEBUG " COPTFLAGS="-std=gnu11 -Wall > -funroll-all-loops -O3 -DNDEBUG" CXXOPTFLAGS="-std=gnu++14 -Wall > -funroll-all-loops -O3 -DNDEBUG " FCFLAGS="-Wall -funroll-all-loops > -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime > -Wno-unused-function -O3 -DNDEBUG" F90FLAGS="-Wall -funroll-all-loops > -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime > -Wno-unused-function -O3 -DNDEBUG" FOPTFLAGS="-Wall -funroll-all-loops > -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime > -Wno-unused-function -O3 -DNDEBUG" > [1]PETSC ERROR: #1 PetscLogNestedTreePrint() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:330 > [1]PETSC ERROR: #2 PetscLogNestedTreePrint() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 > [1]PETSC ERROR: #3 PetscLogNestedTreePrint() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 > [1]PETSC ERROR: #4 PetscLogNestedTreePrintTop() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 > [1]PETSC ERROR: #5 PetscLogHandlerView_Nested_XML() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 > [1]PETSC ERROR: #6 PetscLogHandlerView_Nested() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 > [1]PETSC ERROR: #7 PetscLogHandlerView() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 > [1]PETSC ERROR: #8 PetscLogView() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/plog.c:2040 > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD > with errorcode 98. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > [cid:ii_196725d1e2a809852191] > dr. ir. Christiaan Klaij > | senior researcher | Research & Development | CFD Development > T +31 317 49 33 44 | C.Klaij at marin.nl > >> C.Klaij at marin.nl>> >> C.Klaij at marin.nl>>> >> C.Klaij at marin.nl>> >> C.Klaij at marin.nl>>>> | > https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6g8TwMMcw$ > < > https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM8tSNH1g$ > > > [Facebook]< > https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMcVCZ9hk$ > > > [LinkedIn]< > https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMIDBZW7k$ > > > [YouTube]< > https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMVKWos24$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ > < > https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgMsu6hhA$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ > < > https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgMsu6hhA$ > > > > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Thu May 1 03:23:34 2025 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Thu, 1 May 2025 08:23:34 +0000 Subject: [petsc-users] problem with nested logging In-Reply-To: References: Message-ID: Was the rewritting by Toby done somewhere between petsc 3.19.4 (no problem) and 3.23.4 (problem)? Chris ________________________________________ From: Stefano Zampini Sent: Thursday, May 1, 2025 9:12 AM To: Klaij, Christiaan Cc: Randall Mackie; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging If I look at the code for PetscLogHandlerEventBegin_Default, there are checks in place to see if the event is collectively called (see below) Can you make sure the Nested logging system has the same checks? It should, but double check since the code has been largely rewritten by Toby some time ago; to check it should be as easy as writing a code that calls a collective event on a single process and a debug version of petsc should complain if (rank ==0) LogEventBegin() <-this should call MPIU_Allreduce, but other processes will not, thus hang If it does not complain, then the error must come from how the logic in LogView works, and from how it traverses the various events (my guess: processed in a different order from different processes). Without a reproducer, it is hard to understand what's going on static PetscErrorCode PetscLogHandlerEventBegin_Default(PetscLogHandler h, PetscLogEvent event, PetscObject o1, PetscObject o2, PetscObject o3, PetscObject o4) { PetscLogHandler_Default def = (PetscLogHandler_Default)h->data; PetscEventPerfInfo *event_perf_info = NULL; PetscLogEventInfo event_info; PetscLogDouble time; PetscLogState state; PetscLogStage stage; PetscFunctionBegin; PetscCall(PetscLogHandlerGetState(h, &state)); if (PetscDefined(USE_DEBUG)) { PetscCall(PetscLogStateEventGetInfo(state, event, &event_info)); if (PetscUnlikely(o1)) PetscValidHeader(o1, 3); if (PetscUnlikely(o2)) PetscValidHeader(o2, 4); if (PetscUnlikely(o3)) PetscValidHeader(o3, 5); if (PetscUnlikely(o4)) PetscValidHeader(o4, 6); if (event_info.collective && o1) { PetscInt64 b1[2], b2[2]; b1[0] = -o1->cidx; b1[1] = o1->cidx; PetscCallMPI(MPIU_Allreduce(b1, b2, 2, MPIU_INT64, MPI_MAX, PetscObjectComm(o1))); PetscCheck(-b2[0] == b2[1], PETSC_COMM_SELF, PETSC_ERR_PLIB, "Collective event %s not called collectively %" PetscInt64_FMT " != %" PetscInt64_FMT, event_info.name, -b2[0], b2[1]); } } /* Synchronization */ PetscCall(PetscLogHandlerEventSync_Default(h, event, PetscObjectComm(o1))); Il giorno gio 1 mag 2025 alle ore 09:56 Klaij, Christiaan > ha scritto: I've tried with -log_sync, no complaints whatsoever, but the error is still there... Chris ________________________________________ From: Stefano Zampini > Sent: Tuesday, April 29, 2025 6:12 PM To: Randall Mackie Cc: Klaij, Christiaan; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Can you try using -log_sync ? This should check every entry/exit points of logged Events and complain if something is not collectively called Stefano On Tue, Apr 29, 2025, 18:21 Randall Mackie >> wrote: ah okay, I missed that this was found using openmpi. then it?s probably not the same issue we had. I can?t remember in which version it was fixed (I?m away from my work computer)?.I do know in our case openmpi and the latest Intel One API work fine. Randy On Apr 29, 2025, at 8:58?AM, Klaij, Christiaan >> wrote: Well, the error below only shows-up thanks to openmpi and gnu compilers. With the intel mpi and compilers it just hangs (tried oneapi 2023.1.0). In which version was that bug fixed? Chris ________________________________________ dr. ir.???? Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44 | C.Klaij at marin.nl> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!bpHc3HAqKpoVaOEwsiCMvj6wVV_IPsTf917-ZAkFvGJ4x6txXfCtS14X188Bd8xnTPwkEMlrAwF0wjyFde5T_gM$ From: Randall Mackie >> Sent: Tuesday, April 29, 2025 3:33 PM To: Klaij, Christiaan Cc: Matthew Knepley; petsc-users at mcs.anl.gov>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from rlmackie862 at gmail.com>. Learn why this is important> We had a similar issue last year that we eventually tracked down to a bug in Intel MPI AllReduce, which was around the same version you are using. Can you try a different MPI or the latest Intel One API and see if your error clears? Randy On Tue, Apr 29, 2025 at 8:17?AM Klaij, Christiaan via petsc-users >>>> wrote: I don't think so, we have tracing in place to detect mismatches. But as soon as I switch the tracing on, the error disappears... Same if I add a counter or print statements before and after EventBegin/End. Looks like a memory corruption problem, maybe nothing to do with petsc despite the error message. Chris ________________________________________ From: Matthew Knepley >>>> Sent: Tuesday, April 29, 2025 1:50 PM To: Klaij, Christiaan Cc: Junchao Zhang; petsc-users at mcs.anl.gov>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging On Tue, Apr 29, 2025 at 6:50?AM Klaij, Christiaan >>>>>>>> wrote: Here's a slightly better error message, obtained --with-debugging=1 Is it possible that you have a mismatched EventBegin()/EventEnd() in your code? That could be why we cannot reproduce it here. Thanks, Matt [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: MPIU_Allreduce() called in different locations (code lines) on different processors [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjggvQAzPU$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 [0]PETSC ERROR: ./refresco with 2 MPI process(es) and PETSC_ARCH on marclus3login2 by cklaij Tue Apr 29 12:43:54 2025 [0]PETSC ERROR: Configure options: --prefix=/home/cklaij/ReFRESCO/trunk/install/extLibs --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 --with-debugging=1 --download-superlu_dist=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgVgVAJPM$ --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 --download-parmetis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgo2JWTO4$ --download-metis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgX9ZMYJA$ --with-packages-build-dir=/home/cklaij/ReFRESCO/trunk/build-libs/superbuild --with-ssl=0 --with-shared-libraries=1 [0]PETSC ERROR: #1 PetscLogNestedTreePrintLine() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:289 [0]PETSC ERROR: #2 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:379 [0]PETSC ERROR: #3 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [0]PETSC ERROR: #4 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [0]PETSC ERROR: #5 PetscLogNestedTreePrintTop() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 [0]PETSC ERROR: #6 PetscLogHandlerView_Nested_XML() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 [0]PETSC ERROR: #7 PetscLogHandlerView_Nested() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 [0]PETSC ERROR: #8 PetscLogHandlerView() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 [0]PETSC ERROR: #9 PetscLogView() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/plog.c:2040 [0]PETSC ERROR: #10 /home/cklaij/ReFRESCO/trunk/Code/src/petsc_include_impl.F90:130 ________________________________________ [cid:ii_19681617e7812ff9cfc1] dr. ir. Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44 | C.Klaij at marin.nl>>>>>>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgJooxJhg$ [Facebook] [LinkedIn] [YouTube] From: Klaij, Christiaan >>>>>>>> Sent: Monday, April 28, 2025 3:53 PM To: Matthew Knepley Cc: Junchao Zhang; petsc-users at mcs.anl.gov>>>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Bisecting would be quite hard, it's not just the petsc version that changed, also other libs, compilers, even os components. Chris ________________________________________ From: Matthew Knepley >>>>>>>> Sent: Monday, April 28, 2025 3:06 PM To: Klaij, Christiaan Cc: Junchao Zhang; petsc-users at mcs.anl.gov>>>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from knepley at gmail.com>>>>>>>. Learn why this is important On Mon, Apr 28, 2025 at 8:45?AM Klaij, Christiaan via petsc-users >>>>>>>>>>>>>>>> wrote: I've tried adding a nested log viewer to src/snes/tutorials/ex70.c, but it does not replicate the problem and works fine. Perhaps it is related to fortran, since the manualpage of PetscLogNestedBegin says "No fortran support" (why? we've been using it in fortran ever since). Therefore I've tried adding it to src/snes/ex5f90.F90 and that also works fine. It seems I cannot replicate the problem in a small example, unfortunately. We cannot replicate it here. Is there a chance you could bisect to see what change is responsible? Thanks, Matt Chris ________________________________________ From: Junchao Zhang >>>>>>>>>>>>>>>> Sent: Saturday, April 26, 2025 3:51 PM To: Klaij, Christiaan Cc: petsc-users at mcs.anl.gov>>>>>>>>>>>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from junchao.zhang at gmail.com>>>>>>>>>>>>>>>. Learn why this is important Toby (Cc'ed) might know it. Or could you provide an example? --Junchao Zhang On Fri, Apr 25, 2025 at 3:31?AM Klaij, Christiaan via petsc-users >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: We recently upgraded from 3.19.4 to 3.22.4 but face the problem below with the nested logging. Any ideas? Chris [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: General MPI error [1]PETSC ERROR: MPI error 1 MPI_ERR_BUFFER: invalid buffer pointer [1]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gIT68pbk$ for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 [1]PETSC ERROR: refresco with 2 MPI process(es) and PETSC_ARCH on marclus3login2 by jwindt Fri Apr 25 08:52:30 2025 [1]PETSC ERROR: Configure options: --prefix=/home/jwindt/cmake_builds/refresco/install-libs-gnu --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 --with-debugging=0 --download-superlu_dist=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6grH5BbeU$ --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 --download-parmetis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gw4-tEtY$ --download-metis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gHq4uYiY$ --with-packages-build-dir=/home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild --with-ssl=0 --with-shared-libraries=1 CFLAGS="-std=gnu11 -Wall -funroll-all-loops -O3 -DNDEBUG" CXXFLAGS="-std=gnu++14 -Wall -funroll-all-loops -O3 -DNDEBUG " COPTFLAGS="-std=gnu11 -Wall -funroll-all-loops -O3 -DNDEBUG" CXXOPTFLAGS="-std=gnu++14 -Wall -funroll-all-loops -O3 -DNDEBUG " FCFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" F90FLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" FOPTFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" [1]PETSC ERROR: #1 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:330 [1]PETSC ERROR: #2 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [1]PETSC ERROR: #3 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [1]PETSC ERROR: #4 PetscLogNestedTreePrintTop() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 [1]PETSC ERROR: #5 PetscLogHandlerView_Nested_XML() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 [1]PETSC ERROR: #6 PetscLogHandlerView_Nested() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 [1]PETSC ERROR: #7 PetscLogHandlerView() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 [1]PETSC ERROR: #8 PetscLogView() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/plog.c:2040 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 98. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [cid:ii_196725d1e2a809852191] dr. ir. Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44 | C.Klaij at marin.nl>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6g8TwMMcw$ [Facebook] [LinkedIn] [YouTube] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ -- Stefano From C.Klaij at marin.nl Thu May 1 03:38:49 2025 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Thu, 1 May 2025 08:38:49 +0000 Subject: [petsc-users] problem with nested logging In-Reply-To: References: Message-ID: The checks seem to be in place: I do get a PETSC ERROR when I add a log event on rank 0 as you suggested. Another thought: in between the log events pairs, I also have calls to PetscLogFlops, perhaps that plays a role somehow. Chris ________________________________________ From: Klaij, Christiaan Sent: Thursday, May 1, 2025 10:23 AM To: Stefano Zampini Cc: Randall Mackie; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Was the rewritting by Toby done somewhere between petsc 3.19.4 (no problem) and 3.23.4 (problem)? Chris ________________________________________ From: Stefano Zampini Sent: Thursday, May 1, 2025 9:12 AM To: Klaij, Christiaan Cc: Randall Mackie; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging If I look at the code for PetscLogHandlerEventBegin_Default, there are checks in place to see if the event is collectively called (see below) Can you make sure the Nested logging system has the same checks? It should, but double check since the code has been largely rewritten by Toby some time ago; to check it should be as easy as writing a code that calls a collective event on a single process and a debug version of petsc should complain if (rank ==0) LogEventBegin() <-this should call MPIU_Allreduce, but other processes will not, thus hang If it does not complain, then the error must come from how the logic in LogView works, and from how it traverses the various events (my guess: processed in a different order from different processes). Without a reproducer, it is hard to understand what's going on static PetscErrorCode PetscLogHandlerEventBegin_Default(PetscLogHandler h, PetscLogEvent event, PetscObject o1, PetscObject o2, PetscObject o3, PetscObject o4) { PetscLogHandler_Default def = (PetscLogHandler_Default)h->data; PetscEventPerfInfo *event_perf_info = NULL; PetscLogEventInfo event_info; PetscLogDouble time; PetscLogState state; PetscLogStage stage; PetscFunctionBegin; PetscCall(PetscLogHandlerGetState(h, &state)); if (PetscDefined(USE_DEBUG)) { PetscCall(PetscLogStateEventGetInfo(state, event, &event_info)); if (PetscUnlikely(o1)) PetscValidHeader(o1, 3); if (PetscUnlikely(o2)) PetscValidHeader(o2, 4); if (PetscUnlikely(o3)) PetscValidHeader(o3, 5); if (PetscUnlikely(o4)) PetscValidHeader(o4, 6); if (event_info.collective && o1) { PetscInt64 b1[2], b2[2]; b1[0] = -o1->cidx; b1[1] = o1->cidx; PetscCallMPI(MPIU_Allreduce(b1, b2, 2, MPIU_INT64, MPI_MAX, PetscObjectComm(o1))); PetscCheck(-b2[0] == b2[1], PETSC_COMM_SELF, PETSC_ERR_PLIB, "Collective event %s not called collectively %" PetscInt64_FMT " != %" PetscInt64_FMT, event_info.name, -b2[0], b2[1]); } } /* Synchronization */ PetscCall(PetscLogHandlerEventSync_Default(h, event, PetscObjectComm(o1))); Il giorno gio 1 mag 2025 alle ore 09:56 Klaij, Christiaan > ha scritto: I've tried with -log_sync, no complaints whatsoever, but the error is still there... Chris ________________________________________ From: Stefano Zampini > Sent: Tuesday, April 29, 2025 6:12 PM To: Randall Mackie Cc: Klaij, Christiaan; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Can you try using -log_sync ? This should check every entry/exit points of logged Events and complain if something is not collectively called Stefano On Tue, Apr 29, 2025, 18:21 Randall Mackie >> wrote: ah okay, I missed that this was found using openmpi. then it?s probably not the same issue we had. I can?t remember in which version it was fixed (I?m away from my work computer)?.I do know in our case openmpi and the latest Intel One API work fine. Randy On Apr 29, 2025, at 8:58?AM, Klaij, Christiaan >> wrote: Well, the error below only shows-up thanks to openmpi and gnu compilers. With the intel mpi and compilers it just hangs (tried oneapi 2023.1.0). In which version was that bug fixed? Chris ________________________________________ dr. ir.???? Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44 | C.Klaij at marin.nl> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cLB7VUYhQG17LeyP77ukM-C-LpCMiNkoY1KIHTQNNQIFUd--KmP_pHV9JNNt17iBn3dVJc6G2jHaYPgbpPwjvDg$ From: Randall Mackie >> Sent: Tuesday, April 29, 2025 3:33 PM To: Klaij, Christiaan Cc: Matthew Knepley; petsc-users at mcs.anl.gov>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from rlmackie862 at gmail.com>. Learn why this is important> We had a similar issue last year that we eventually tracked down to a bug in Intel MPI AllReduce, which was around the same version you are using. Can you try a different MPI or the latest Intel One API and see if your error clears? Randy On Tue, Apr 29, 2025 at 8:17?AM Klaij, Christiaan via petsc-users >>>> wrote: I don't think so, we have tracing in place to detect mismatches. But as soon as I switch the tracing on, the error disappears... Same if I add a counter or print statements before and after EventBegin/End. Looks like a memory corruption problem, maybe nothing to do with petsc despite the error message. Chris ________________________________________ From: Matthew Knepley >>>> Sent: Tuesday, April 29, 2025 1:50 PM To: Klaij, Christiaan Cc: Junchao Zhang; petsc-users at mcs.anl.gov>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging On Tue, Apr 29, 2025 at 6:50?AM Klaij, Christiaan >>>>>>>> wrote: Here's a slightly better error message, obtained --with-debugging=1 Is it possible that you have a mismatched EventBegin()/EventEnd() in your code? That could be why we cannot reproduce it here. Thanks, Matt [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: MPIU_Allreduce() called in different locations (code lines) on different processors [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjggvQAzPU$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 [0]PETSC ERROR: ./refresco with 2 MPI process(es) and PETSC_ARCH on marclus3login2 by cklaij Tue Apr 29 12:43:54 2025 [0]PETSC ERROR: Configure options: --prefix=/home/cklaij/ReFRESCO/trunk/install/extLibs --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 --with-debugging=1 --download-superlu_dist=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgVgVAJPM$ --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 --download-parmetis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgo2JWTO4$ --download-metis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgX9ZMYJA$ --with-packages-build-dir=/home/cklaij/ReFRESCO/trunk/build-libs/superbuild --with-ssl=0 --with-shared-libraries=1 [0]PETSC ERROR: #1 PetscLogNestedTreePrintLine() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:289 [0]PETSC ERROR: #2 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:379 [0]PETSC ERROR: #3 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [0]PETSC ERROR: #4 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [0]PETSC ERROR: #5 PetscLogNestedTreePrintTop() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 [0]PETSC ERROR: #6 PetscLogHandlerView_Nested_XML() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 [0]PETSC ERROR: #7 PetscLogHandlerView_Nested() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 [0]PETSC ERROR: #8 PetscLogHandlerView() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 [0]PETSC ERROR: #9 PetscLogView() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/plog.c:2040 [0]PETSC ERROR: #10 /home/cklaij/ReFRESCO/trunk/Code/src/petsc_include_impl.F90:130 ________________________________________ [cid:ii_19681617e7812ff9cfc1] dr. ir. Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44 | C.Klaij at marin.nl>>>>>>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgJooxJhg$ [Facebook] [LinkedIn] [YouTube] From: Klaij, Christiaan >>>>>>>> Sent: Monday, April 28, 2025 3:53 PM To: Matthew Knepley Cc: Junchao Zhang; petsc-users at mcs.anl.gov>>>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Bisecting would be quite hard, it's not just the petsc version that changed, also other libs, compilers, even os components. Chris ________________________________________ From: Matthew Knepley >>>>>>>> Sent: Monday, April 28, 2025 3:06 PM To: Klaij, Christiaan Cc: Junchao Zhang; petsc-users at mcs.anl.gov>>>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from knepley at gmail.com>>>>>>>. Learn why this is important On Mon, Apr 28, 2025 at 8:45?AM Klaij, Christiaan via petsc-users >>>>>>>>>>>>>>>> wrote: I've tried adding a nested log viewer to src/snes/tutorials/ex70.c, but it does not replicate the problem and works fine. Perhaps it is related to fortran, since the manualpage of PetscLogNestedBegin says "No fortran support" (why? we've been using it in fortran ever since). Therefore I've tried adding it to src/snes/ex5f90.F90 and that also works fine. It seems I cannot replicate the problem in a small example, unfortunately. We cannot replicate it here. Is there a chance you could bisect to see what change is responsible? Thanks, Matt Chris ________________________________________ From: Junchao Zhang >>>>>>>>>>>>>>>> Sent: Saturday, April 26, 2025 3:51 PM To: Klaij, Christiaan Cc: petsc-users at mcs.anl.gov>>>>>>>>>>>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from junchao.zhang at gmail.com>>>>>>>>>>>>>>>. Learn why this is important Toby (Cc'ed) might know it. Or could you provide an example? --Junchao Zhang On Fri, Apr 25, 2025 at 3:31?AM Klaij, Christiaan via petsc-users >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: We recently upgraded from 3.19.4 to 3.22.4 but face the problem below with the nested logging. Any ideas? Chris [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: General MPI error [1]PETSC ERROR: MPI error 1 MPI_ERR_BUFFER: invalid buffer pointer [1]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gIT68pbk$ for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 [1]PETSC ERROR: refresco with 2 MPI process(es) and PETSC_ARCH on marclus3login2 by jwindt Fri Apr 25 08:52:30 2025 [1]PETSC ERROR: Configure options: --prefix=/home/jwindt/cmake_builds/refresco/install-libs-gnu --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 --with-debugging=0 --download-superlu_dist=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6grH5BbeU$ --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 --download-parmetis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gw4-tEtY$ --download-metis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gHq4uYiY$ --with-packages-build-dir=/home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild --with-ssl=0 --with-shared-libraries=1 CFLAGS="-std=gnu11 -Wall -funroll-all-loops -O3 -DNDEBUG" CXXFLAGS="-std=gnu++14 -Wall -funroll-all-loops -O3 -DNDEBUG " COPTFLAGS="-std=gnu11 -Wall -funroll-all-loops -O3 -DNDEBUG" CXXOPTFLAGS="-std=gnu++14 -Wall -funroll-all-loops -O3 -DNDEBUG " FCFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" F90FLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" FOPTFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" [1]PETSC ERROR: #1 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:330 [1]PETSC ERROR: #2 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [1]PETSC ERROR: #3 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [1]PETSC ERROR: #4 PetscLogNestedTreePrintTop() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 [1]PETSC ERROR: #5 PetscLogHandlerView_Nested_XML() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 [1]PETSC ERROR: #6 PetscLogHandlerView_Nested() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 [1]PETSC ERROR: #7 PetscLogHandlerView() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 [1]PETSC ERROR: #8 PetscLogView() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/plog.c:2040 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 98. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [cid:ii_196725d1e2a809852191] dr. ir. Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44 | C.Klaij at marin.nl>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6g8TwMMcw$ [Facebook] [LinkedIn] [YouTube] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ -- Stefano From stefano.zampini at gmail.com Thu May 1 03:51:26 2025 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Thu, 1 May 2025 11:51:26 +0300 Subject: [petsc-users] problem with nested logging In-Reply-To: References: Message-ID: Il giorno gio 1 mag 2025 alle ore 11:23 Klaij, Christiaan ha scritto: > Was the rewritting by Toby done somewhere between petsc 3.19.4 (no > problem) and 3.23.4 (problem)? > > If I look at changelogs, I see many PetscLog related changes in doc/changes/320.md, so I presume the answer is yes -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Thu May 1 03:57:08 2025 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Thu, 1 May 2025 11:57:08 +0300 Subject: [petsc-users] problem with nested logging In-Reply-To: References: Message-ID: Il giorno gio 1 mag 2025 alle ore 11:38 Klaij, Christiaan ha scritto: > The checks seem to be in place: I do get a PETSC ERROR when I add a log > event on rank 0 as you suggested. > > Ok, the broken logic may be in LogView then. You can try deactivating some logging by classes and see how the error evolves, maybe using PetscLogClassSetActiveAll. Or, if feasible, commenting out some part of the simulation code > Another thought: in between the log events pairs, I also have calls to > PetscLogFlops, perhaps that plays a role somehow. > > It shouldn't > Chris > > ________________________________________ > From: Klaij, Christiaan > Sent: Thursday, May 1, 2025 10:23 AM > To: Stefano Zampini > Cc: Randall Mackie; PETSc users list; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > Was the rewritting by Toby done somewhere between petsc 3.19.4 (no > problem) and 3.23.4 (problem)? > > Chris > > ________________________________________ > From: Stefano Zampini > Sent: Thursday, May 1, 2025 9:12 AM > To: Klaij, Christiaan > Cc: Randall Mackie; PETSc users list; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > If I look at the code for PetscLogHandlerEventBegin_Default, there are > checks in place to see if the event is collectively called (see below) > Can you make sure the Nested logging system has the same checks? > It should, but double check since the code has been largely rewritten by > Toby some time ago; to check it should be as easy as writing a code that > calls a collective event on a single process and a debug version of petsc > should complain > > if (rank ==0) > LogEventBegin() <-this should call MPIU_Allreduce, but other processes > will not, thus hang > > > If it does not complain, then the error must come from how the logic in > LogView works, and from how it traverses the various events (my guess: > processed in a different order from different processes). Without a > reproducer, it is hard to understand what's going on > > static PetscErrorCode PetscLogHandlerEventBegin_Default(PetscLogHandler h, > PetscLogEvent event, PetscObject o1, PetscObject o2, PetscObject o3, > PetscObject o4) > { > PetscLogHandler_Default def = > (PetscLogHandler_Default)h->data; > PetscEventPerfInfo *event_perf_info = NULL; > PetscLogEventInfo event_info; > PetscLogDouble time; > PetscLogState state; > PetscLogStage stage; > > PetscFunctionBegin; > PetscCall(PetscLogHandlerGetState(h, &state)); > if (PetscDefined(USE_DEBUG)) { > PetscCall(PetscLogStateEventGetInfo(state, event, &event_info)); > if (PetscUnlikely(o1)) PetscValidHeader(o1, 3); > if (PetscUnlikely(o2)) PetscValidHeader(o2, 4); > if (PetscUnlikely(o3)) PetscValidHeader(o3, 5); > if (PetscUnlikely(o4)) PetscValidHeader(o4, 6); > if (event_info.collective && o1) { > PetscInt64 b1[2], b2[2]; > > b1[0] = -o1->cidx; > b1[1] = o1->cidx; > PetscCallMPI(MPIU_Allreduce(b1, b2, 2, MPIU_INT64, MPI_MAX, > PetscObjectComm(o1))); > PetscCheck(-b2[0] == b2[1], PETSC_COMM_SELF, PETSC_ERR_PLIB, > "Collective event %s not called collectively %" PetscInt64_FMT " != %" > PetscInt64_FMT, event_info.name, -b2[0], b2[1]); > } > } > /* Synchronization */ > PetscCall(PetscLogHandlerEventSync_Default(h, event, > PetscObjectComm(o1))); > > > > > Il giorno gio 1 mag 2025 alle ore 09:56 Klaij, Christiaan < > C.Klaij at marin.nl> ha scritto: > I've tried with -log_sync, no complaints whatsoever, but the error is > still there... > > Chris > > ________________________________________ > From: Stefano Zampini stefano.zampini at gmail.com>> > Sent: Tuesday, April 29, 2025 6:12 PM > To: Randall Mackie > Cc: Klaij, Christiaan; PETSc users list; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > Can you try using -log_sync ? This should check every entry/exit points of > logged Events and complain if something is not collectively called > > Stefano > > On Tue, Apr 29, 2025, 18:21 Randall Mackie rlmackie862 at gmail.com> rlmackie862 at gmail.com>>> wrote: > ah okay, I missed that this was found using openmpi. > > then it?s probably not the same issue we had. > > I can?t remember in which version it was fixed (I?m away from my work > computer)?.I do know in our case openmpi and the latest Intel One API work > fine. > > Randy > > On Apr 29, 2025, at 8:58?AM, Klaij, Christiaan C.Klaij at marin.nl>>> > wrote: > > Well, the error below only shows-up thanks to openmpi and gnu compilers. > With the intel mpi and compilers it just hangs (tried oneapi 2023.1.0). In > which version was that bug fixed? > > Chris > > ________________________________________ > > dr. ir. Christiaan Klaij > | senior researcher | Research & Development | > CFD Development > T +31 317 49 33 44 | > C.Klaij at marin.nl C.Klaij at marin.nl>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!crCM5sjjbiW2UA3oXM3tEMAALyn_2tPaKI8PGqZHWnSyRxkWGSkyxCXMSOOX4ahR0lDueyvZM3jiyD8WwsnM8kQGA8tjmMQ$ < > https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cx-ci9iNg$ > > > < > https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cxl-rBbfA$ > > > < > https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cw7FNelIQ$ > > > < > https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6CwmfHQRog$ > > > > From: Randall Mackie >>> > Sent: Tuesday, April 29, 2025 3:33 PM > To: Klaij, Christiaan > Cc: Matthew Knepley; petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov>>; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > You don't often get email from rlmackie862 at gmail.com rlmackie862 at gmail.com> rlmackie862 at gmail.com>>. Learn why this is important< > https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!crCM5sjjbiW2UA3oXM3tEMAALyn_2tPaKI8PGqZHWnSyRxkWGSkyxCXMSOOX4ahR0lDueyvZM3jiyD8WwsnM8kQG_JZIR7c$ < > https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6CwgJDUS5w$ > >> > We had a similar issue last year that we eventually tracked down to a bug > in Intel MPI AllReduce, which was around the same version you are using. > > Can you try a different MPI or the latest Intel One API and see if your > error clears? > > Randy > > On Tue, Apr 29, 2025 at 8:17?AM Klaij, Christiaan via petsc-users < > petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>> wrote: > I don't think so, we have tracing in place to detect mismatches. But as > soon as I switch the tracing on, the error disappears... Same if I add a > counter or print statements before and after EventBegin/End. Looks like a > memory corruption problem, maybe nothing to do with petsc despite the error > message. > > Chris > > ________________________________________ > From: Matthew Knepley knepley at gmail.com> knepley at gmail.com>>>> > Sent: Tuesday, April 29, 2025 1:50 PM > To: Klaij, Christiaan > Cc: Junchao Zhang; petsc-users at mcs.anl.gov >> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > On Tue, Apr 29, 2025 at 6:50?AM Klaij, Christiaan >> C.Klaij at marin.nl>> >> C.Klaij at marin.nl>>>> wrote: > Here's a slightly better error message, obtained --with-debugging=1 > > Is it possible that you have a mismatched EventBegin()/EventEnd() in your > code? That could be why we cannot reproduce it here. > > Thanks, > > Matt > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Petsc has generated inconsistent data > [0]PETSC ERROR: MPIU_Allreduce() called in different locations (code > lines) on different processors > [0]PETSC ERROR: See > https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjggvQAzPU$ > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 > [0]PETSC ERROR: ./refresco with 2 MPI process(es) and PETSC_ARCH on > marclus3login2 by cklaij Tue Apr 29 12:43:54 2025 > [0]PETSC ERROR: Configure options: > --prefix=/home/cklaij/ReFRESCO/trunk/install/extLibs > --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 > --with-debugging=1 --download-superlu_dist= > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgVgVAJPM$ > --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 > --download-parmetis= > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgo2JWTO4$ > --download-metis= > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgX9ZMYJA$ > --with-packages-build-dir=/home/cklaij/ReFRESCO/trunk/build-libs/superbuild > --with-ssl=0 --with-shared-libraries=1 > [0]PETSC ERROR: #1 PetscLogNestedTreePrintLine() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:289 > [0]PETSC ERROR: #2 PetscLogNestedTreePrint() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:379 > [0]PETSC ERROR: #3 PetscLogNestedTreePrint() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 > [0]PETSC ERROR: #4 PetscLogNestedTreePrint() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 > [0]PETSC ERROR: #5 PetscLogNestedTreePrintTop() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 > [0]PETSC ERROR: #6 PetscLogHandlerView_Nested_XML() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 > [0]PETSC ERROR: #7 PetscLogHandlerView_Nested() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 > [0]PETSC ERROR: #8 PetscLogHandlerView() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 > [0]PETSC ERROR: #9 PetscLogView() at > /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/plog.c:2040 > [0]PETSC ERROR: #10 > /home/cklaij/ReFRESCO/trunk/Code/src/petsc_include_impl.F90:130 > > ________________________________________ > [cid:ii_19681617e7812ff9cfc1] > dr. ir. Christiaan Klaij > | senior researcher | Research & Development | CFD Development > T +31 317 49 33 44 | C.Klaij at marin.nl > >> C.Klaij at marin.nl>> >> C.Klaij at marin.nl>>> | > https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgJooxJhg$ > < > https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgpOXjQSM$ > > > [Facebook]< > https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg_3Zt0Pw$ > > > [LinkedIn]< > https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgcpgQdSE$ > > > [YouTube]< > https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgn8qP6_I$ > > > > From: Klaij, Christiaan C.Klaij at marin.nl> C.Klaij at marin.nl> >>> C.Klaij at marin.nl> C.Klaij at marin.nl>>>>> > Sent: Monday, April 28, 2025 3:53 PM > To: Matthew Knepley > Cc: Junchao Zhang; petsc-users at mcs.anl.gov >> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>>; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > Bisecting would be quite hard, it's not just the petsc version that > changed, also other libs, compilers, even os components. > > Chris > > ________________________________________ > From: Matthew Knepley knepley at gmail.com> knepley at gmail.com>>> >> knepley at gmail.com >>>> > Sent: Monday, April 28, 2025 3:06 PM > To: Klaij, Christiaan > Cc: Junchao Zhang; petsc-users at mcs.anl.gov >> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>>; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > You don't often get email from knepley at gmail.com >> knepley at gmail.com >> knepley at gmail.com> >> knepley at gmail.com>>>. Learn why this is > important< > https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgcKEJXRA$ > > > On Mon, Apr 28, 2025 at 8:45?AM Klaij, Christiaan via petsc-users < > petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>>>> wrote: > I've tried adding a nested log viewer to src/snes/tutorials/ex70.c, > but it does not replicate the problem and works fine. > > Perhaps it is related to fortran, since the manualpage of > PetscLogNestedBegin says "No fortran support" (why? we've been > using it in fortran ever since). > > Therefore I've tried adding it to src/snes/ex5f90.F90 and that > also works fine. It seems I cannot replicate the problem in a > small example, unfortunately. > > We cannot replicate it here. Is there a chance you could bisect to see > what change is responsible? > > Thanks, > > Matt > > Chris > > ________________________________________ > From: Junchao Zhang junchao.zhang at gmail.com> junchao.zhang at gmail.com>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>>>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>>>>>> > Sent: Saturday, April 26, 2025 3:51 PM > To: Klaij, Christiaan > Cc: petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>>>; Isaac, Toby > Subject: Re: [petsc-users] problem with nested logging > > You don't often get email from junchao.zhang at gmail.com junchao.zhang at gmail.com> junchao.zhang at gmail.com>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>>>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>> junchao.zhang at gmail.com> junchao.zhang at gmail.com>>>>>. Learn why this is important< > https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gVgt1dmE$ > > > Toby (Cc'ed) might know it. Or could you provide an example? > > --Junchao Zhang > > > On Fri, Apr 25, 2025 at 3:31?AM Klaij, Christiaan via petsc-users < > petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov> petsc-users at mcs.anl.gov petsc-users at mcs.anl.gov>>>>>> wrote: > We recently upgraded from 3.19.4 to 3.22.4 but face the problem below with > the nested logging. Any ideas? > > Chris > > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: General MPI error > [1]PETSC ERROR: MPI error 1 MPI_ERR_BUFFER: invalid buffer pointer > [1]PETSC ERROR: See > https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gIT68pbk$ > < > https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM3UaqHzc$> > for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 > [1]PETSC ERROR: refresco with 2 MPI process(es) and PETSC_ARCH on > marclus3login2 by jwindt Fri Apr 25 08:52:30 2025 > [1]PETSC ERROR: Configure options: > --prefix=/home/jwindt/cmake_builds/refresco/install-libs-gnu > --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 > --with-debugging=0 --download-superlu_dist= > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6grH5BbeU$ > < > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM21-2D-o$> > --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 > --download-parmetis= > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gw4-tEtY$ > < > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMW0lYHko$> > --download-metis= > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gHq4uYiY$ > < > https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMbSrIiUg$> > --with-packages-build-dir=/home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild > --with-ssl=0 --with-shared-libraries=1 CFLAGS="-std=gnu11 -Wall > -funroll-all-loops -O3 -DNDEBUG" CXXFLAGS="-std=gnu++14 -Wall > -funroll-all-loops -O3 -DNDEBUG " COPTFLAGS="-std=gnu11 -Wall > -funroll-all-loops -O3 -DNDEBUG" CXXOPTFLAGS="-std=gnu++14 -Wall > -funroll-all-loops -O3 -DNDEBUG " FCFLAGS="-Wall -funroll-all-loops > -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime > -Wno-unused-function -O3 -DNDEBUG" F90FLAGS="-Wall -funroll-all-loops > -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime > -Wno-unused-function -O3 -DNDEBUG" FOPTFLAGS="-Wall -funroll-all-loops > -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime > -Wno-unused-function -O3 -DNDEBUG" > [1]PETSC ERROR: #1 PetscLogNestedTreePrint() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:330 > [1]PETSC ERROR: #2 PetscLogNestedTreePrint() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 > [1]PETSC ERROR: #3 PetscLogNestedTreePrint() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 > [1]PETSC ERROR: #4 PetscLogNestedTreePrintTop() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 > [1]PETSC ERROR: #5 PetscLogHandlerView_Nested_XML() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 > [1]PETSC ERROR: #6 PetscLogHandlerView_Nested() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 > [1]PETSC ERROR: #7 PetscLogHandlerView() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 > [1]PETSC ERROR: #8 PetscLogView() at > /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/plog.c:2040 > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD > with errorcode 98. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > [cid:ii_196725d1e2a809852191] > dr. ir. Christiaan Klaij > | senior researcher | Research & Development | CFD Development > T +31 317 49 33 44 | C.Klaij at marin.nl > >> C.Klaij at marin.nl>> >> C.Klaij at marin.nl>>> >> C.Klaij at marin.nl>> >> C.Klaij at marin.nl>>>> >> C.Klaij at marin.nl>> >> C.Klaij at marin.nl>>> >> C.Klaij at marin.nl>> >> C.Klaij at marin.nl>>>>> | > https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6g8TwMMcw$ > < > https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM8tSNH1g$ > > > [Facebook]< > https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMcVCZ9hk$ > > > [LinkedIn]< > https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMIDBZW7k$ > > > [YouTube]< > https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMVKWos24$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ > < > https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgMsu6hhA$ > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ > < > https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgMsu6hhA$ > > > > > > -- > Stefano > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Thu May 1 08:06:22 2025 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Thu, 1 May 2025 13:06:22 +0000 Subject: [petsc-users] problem with nested logging In-Reply-To: References: Message-ID: If I deactivate all the calls to PetscLogEventBegin/End in the simulation code, the error does not show-up. But since there are more than 2500 events, it's impossible to do them one-by-one, especially since the error shows-up at random and requires a number of cases and repetitions. Unfortunately, I'm running out of time and budget. I will retry once we get Rocky Linux 9 and the latest Intel compilers. Chris ________________________________________ From: Stefano Zampini Sent: Thursday, May 1, 2025 10:57 AM To: Klaij, Christiaan Cc: Randall Mackie; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Il giorno gio 1 mag 2025 alle ore 11:38 Klaij, Christiaan > ha scritto: The checks seem to be in place: I do get a PETSC ERROR when I add a log event on rank 0 as you suggested. Ok, the broken logic may be in LogView then. You can try deactivating some logging by classes and see how the error evolves, maybe using PetscLogClassSetActiveAll. Or, if feasible, commenting out some part of the simulation code Another thought: in between the log events pairs, I also have calls to PetscLogFlops, perhaps that plays a role somehow. It shouldn't Chris ________________________________________ From: Klaij, Christiaan > Sent: Thursday, May 1, 2025 10:23 AM To: Stefano Zampini Cc: Randall Mackie; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Was the rewritting by Toby done somewhere between petsc 3.19.4 (no problem) and 3.23.4 (problem)? Chris ________________________________________ From: Stefano Zampini > Sent: Thursday, May 1, 2025 9:12 AM To: Klaij, Christiaan Cc: Randall Mackie; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging If I look at the code for PetscLogHandlerEventBegin_Default, there are checks in place to see if the event is collectively called (see below) Can you make sure the Nested logging system has the same checks? It should, but double check since the code has been largely rewritten by Toby some time ago; to check it should be as easy as writing a code that calls a collective event on a single process and a debug version of petsc should complain if (rank ==0) LogEventBegin() <-this should call MPIU_Allreduce, but other processes will not, thus hang If it does not complain, then the error must come from how the logic in LogView works, and from how it traverses the various events (my guess: processed in a different order from different processes). Without a reproducer, it is hard to understand what's going on static PetscErrorCode PetscLogHandlerEventBegin_Default(PetscLogHandler h, PetscLogEvent event, PetscObject o1, PetscObject o2, PetscObject o3, PetscObject o4) { PetscLogHandler_Default def = (PetscLogHandler_Default)h->data; PetscEventPerfInfo *event_perf_info = NULL; PetscLogEventInfo event_info; PetscLogDouble time; PetscLogState state; PetscLogStage stage; PetscFunctionBegin; PetscCall(PetscLogHandlerGetState(h, &state)); if (PetscDefined(USE_DEBUG)) { PetscCall(PetscLogStateEventGetInfo(state, event, &event_info)); if (PetscUnlikely(o1)) PetscValidHeader(o1, 3); if (PetscUnlikely(o2)) PetscValidHeader(o2, 4); if (PetscUnlikely(o3)) PetscValidHeader(o3, 5); if (PetscUnlikely(o4)) PetscValidHeader(o4, 6); if (event_info.collective && o1) { PetscInt64 b1[2], b2[2]; b1[0] = -o1->cidx; b1[1] = o1->cidx; PetscCallMPI(MPIU_Allreduce(b1, b2, 2, MPIU_INT64, MPI_MAX, PetscObjectComm(o1))); PetscCheck(-b2[0] == b2[1], PETSC_COMM_SELF, PETSC_ERR_PLIB, "Collective event %s not called collectively %" PetscInt64_FMT " != %" PetscInt64_FMT, event_info.name, -b2[0], b2[1]); } } /* Synchronization */ PetscCall(PetscLogHandlerEventSync_Default(h, event, PetscObjectComm(o1))); Il giorno gio 1 mag 2025 alle ore 09:56 Klaij, Christiaan >> ha scritto: I've tried with -log_sync, no complaints whatsoever, but the error is still there... Chris ________________________________________ From: Stefano Zampini >> Sent: Tuesday, April 29, 2025 6:12 PM To: Randall Mackie Cc: Klaij, Christiaan; PETSc users list; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Can you try using -log_sync ? This should check every entry/exit points of logged Events and complain if something is not collectively called Stefano On Tue, Apr 29, 2025, 18:21 Randall Mackie >>>> wrote: ah okay, I missed that this was found using openmpi. then it?s probably not the same issue we had. I can?t remember in which version it was fixed (I?m away from my work computer)?.I do know in our case openmpi and the latest Intel One API work fine. Randy On Apr 29, 2025, at 8:58?AM, Klaij, Christiaan >>>> wrote: Well, the error below only shows-up thanks to openmpi and gnu compilers. With the intel mpi and compilers it just hangs (tried oneapi 2023.1.0). In which version was that bug fixed? Chris ________________________________________ dr. ir. Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44 | C.Klaij at marin.nl>>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!ZpaWiQzC8-ICOxsOB6S2Oje7LO1aN83c9KBv0AjVb241voYZ1PrPHjGj5Pjk-lRja8JpDxpgrp8y3fiYrepzcmY$ From: Randall Mackie >>>> Sent: Tuesday, April 29, 2025 3:33 PM To: Klaij, Christiaan Cc: Matthew Knepley; petsc-users at mcs.anl.gov>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from rlmackie862 at gmail.com>>>. Learn why this is important> We had a similar issue last year that we eventually tracked down to a bug in Intel MPI AllReduce, which was around the same version you are using. Can you try a different MPI or the latest Intel One API and see if your error clears? Randy On Tue, Apr 29, 2025 at 8:17?AM Klaij, Christiaan via petsc-users >>>>>>>> wrote: I don't think so, we have tracing in place to detect mismatches. But as soon as I switch the tracing on, the error disappears... Same if I add a counter or print statements before and after EventBegin/End. Looks like a memory corruption problem, maybe nothing to do with petsc despite the error message. Chris ________________________________________ From: Matthew Knepley >>>>>>>> Sent: Tuesday, April 29, 2025 1:50 PM To: Klaij, Christiaan Cc: Junchao Zhang; petsc-users at mcs.anl.gov>>>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging On Tue, Apr 29, 2025 at 6:50?AM Klaij, Christiaan >>>>>>>>>>>>>>>> wrote: Here's a slightly better error message, obtained --with-debugging=1 Is it possible that you have a mismatched EventBegin()/EventEnd() in your code? That could be why we cannot reproduce it here. Thanks, Matt [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: MPIU_Allreduce() called in different locations (code lines) on different processors [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjggvQAzPU$ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 [0]PETSC ERROR: ./refresco with 2 MPI process(es) and PETSC_ARCH on marclus3login2 by cklaij Tue Apr 29 12:43:54 2025 [0]PETSC ERROR: Configure options: --prefix=/home/cklaij/ReFRESCO/trunk/install/extLibs --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 --with-debugging=1 --download-superlu_dist=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgVgVAJPM$ --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 --download-parmetis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgo2JWTO4$ --download-metis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgX9ZMYJA$ --with-packages-build-dir=/home/cklaij/ReFRESCO/trunk/build-libs/superbuild --with-ssl=0 --with-shared-libraries=1 [0]PETSC ERROR: #1 PetscLogNestedTreePrintLine() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:289 [0]PETSC ERROR: #2 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:379 [0]PETSC ERROR: #3 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [0]PETSC ERROR: #4 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [0]PETSC ERROR: #5 PetscLogNestedTreePrintTop() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 [0]PETSC ERROR: #6 PetscLogHandlerView_Nested_XML() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 [0]PETSC ERROR: #7 PetscLogHandlerView_Nested() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 [0]PETSC ERROR: #8 PetscLogHandlerView() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 [0]PETSC ERROR: #9 PetscLogView() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/plog.c:2040 [0]PETSC ERROR: #10 /home/cklaij/ReFRESCO/trunk/Code/src/petsc_include_impl.F90:130 ________________________________________ [cid:ii_19681617e7812ff9cfc1] dr. ir. Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44 | C.Klaij at marin.nl>>>>>>>>>>>>>>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgJooxJhg$ [Facebook] [LinkedIn] [YouTube] From: Klaij, Christiaan >>>>>>>>>>>>>>>> Sent: Monday, April 28, 2025 3:53 PM To: Matthew Knepley Cc: Junchao Zhang; petsc-users at mcs.anl.gov>>>>>>>>>>>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging Bisecting would be quite hard, it's not just the petsc version that changed, also other libs, compilers, even os components. Chris ________________________________________ From: Matthew Knepley >>>>>>>>>>>>>>>> Sent: Monday, April 28, 2025 3:06 PM To: Klaij, Christiaan Cc: Junchao Zhang; petsc-users at mcs.anl.gov>>>>>>>>>>>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from knepley at gmail.com>>>>>>>>>>>>>>>. Learn why this is important On Mon, Apr 28, 2025 at 8:45?AM Klaij, Christiaan via petsc-users >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: I've tried adding a nested log viewer to src/snes/tutorials/ex70.c, but it does not replicate the problem and works fine. Perhaps it is related to fortran, since the manualpage of PetscLogNestedBegin says "No fortran support" (why? we've been using it in fortran ever since). Therefore I've tried adding it to src/snes/ex5f90.F90 and that also works fine. It seems I cannot replicate the problem in a small example, unfortunately. We cannot replicate it here. Is there a chance you could bisect to see what change is responsible? Thanks, Matt Chris ________________________________________ From: Junchao Zhang >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: Saturday, April 26, 2025 3:51 PM To: Klaij, Christiaan Cc: petsc-users at mcs.anl.gov>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>; Isaac, Toby Subject: Re: [petsc-users] problem with nested logging You don't often get email from junchao.zhang at gmail.com>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>. Learn why this is important Toby (Cc'ed) might know it. Or could you provide an example? --Junchao Zhang On Fri, Apr 25, 2025 at 3:31?AM Klaij, Christiaan via petsc-users >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: We recently upgraded from 3.19.4 to 3.22.4 but face the problem below with the nested logging. Any ideas? Chris [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: General MPI error [1]PETSC ERROR: MPI error 1 MPI_ERR_BUFFER: invalid buffer pointer [1]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gIT68pbk$ for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025 [1]PETSC ERROR: refresco with 2 MPI process(es) and PETSC_ARCH on marclus3login2 by jwindt Fri Apr 25 08:52:30 2025 [1]PETSC ERROR: Configure options: --prefix=/home/jwindt/cmake_builds/refresco/install-libs-gnu --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 --with-debugging=0 --download-superlu_dist=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6grH5BbeU$ --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 --download-parmetis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gw4-tEtY$ --download-metis=https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gHq4uYiY$ --with-packages-build-dir=/home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild --with-ssl=0 --with-shared-libraries=1 CFLAGS="-std=gnu11 -Wall -funroll-all-loops -O3 -DNDEBUG" CXXFLAGS="-std=gnu++14 -Wall -funroll-all-loops -O3 -DNDEBUG " COPTFLAGS="-std=gnu11 -Wall -funroll-all-loops -O3 -DNDEBUG" CXXOPTFLAGS="-std=gnu++14 -Wall -funroll-all-loops -O3 -DNDEBUG " FCFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" F90FLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" FOPTFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" [1]PETSC ERROR: #1 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:330 [1]PETSC ERROR: #2 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [1]PETSC ERROR: #3 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384 [1]PETSC ERROR: #4 PetscLogNestedTreePrintTop() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420 [1]PETSC ERROR: #5 PetscLogHandlerView_Nested_XML() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443 [1]PETSC ERROR: #6 PetscLogHandlerView_Nested() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405 [1]PETSC ERROR: #7 PetscLogHandlerView() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342 [1]PETSC ERROR: #8 PetscLogView() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/plog.c:2040 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 98. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [cid:ii_196725d1e2a809852191] dr. ir. Christiaan Klaij | senior researcher | Research & Development | CFD Development T +31 317 49 33 44 | C.Klaij at marin.nl>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6g8TwMMcw$ [Facebook] [LinkedIn] [YouTube] -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$ -- Stefano -- Stefano From mmolinos at us.es Mon May 5 02:52:20 2025 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Mon, 5 May 2025 07:52:20 +0000 Subject: [petsc-users] Problems with the null-space removal Message-ID: Dear all, I am working on the resolution of a non-linear problem via SNES with a large null-space (at least 70% of the equations are trivial). In a nutshell I have a mesh with active and non-active nodes, and the non-active nodes introduces the null-space. So far I've dealing with it using NGMRES but I want to use a scheme based on NR. However, despite removing the null-space using PETSc's in-build functions the linear solver (KSP+PC) diverges at the first iteration. However, If I remove the non-active nodes and use the same solver (SNES+KSP+PC), which failed before, now I can solve the non-linear system. Any clue? I?m missing something? Thanks, Miguel From knepley at gmail.com Mon May 5 03:44:05 2025 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 5 May 2025 04:44:05 -0400 Subject: [petsc-users] Problems with the null-space removal In-Reply-To: References: Message-ID: On Mon, May 5, 2025 at 3:52?AM MIGUEL MOLINOS PEREZ wrote: > Dear all, > > I am working on the resolution of a non-linear problem via SNES with a > large null-space (at least 70% of the equations are trivial). In a nutshell > I have a mesh with active and non-active nodes, and the non-active nodes > introduces the null-space. > > So far I've dealing with it using NGMRES but I want to use a scheme based > on NR. However, despite removing the null-space using PETSc's in-build > functions the linear solver (KSP+PC) diverges at the first iteration. > However, If I remove the non-active nodes and use the same solver > (SNES+KSP+PC), which failed before, now I can solve the non-linear system. > > Any clue? I?m missing something? > The most likely thing is that the nullspace is slightly wrong, or the convergence test is not accounting for the nullspace somehow. However, since so many equations are inactive, I would consider projecting the problem. I might either 1) Select a subset of the mesh to phrase the problem on. In you use DMPlex, you could use DMPlexFilter(). 2) Use a constraint to get a subsystem for Newton using SNESVI. The SNESVIRS will project the linear system onto a subspace without the constraints. Thanks, Matt > Thanks, > Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bByLEARdv_XsY4xZu0ezoqCT2bh14450x43ffJDakWXMjru0Po-BcPS5TJl_mDXpjWZaJkliCDj07YjYyLjm$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Mon May 5 13:08:21 2025 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Mon, 5 May 2025 18:08:21 +0000 Subject: [petsc-users] Problems with the null-space removal In-Reply-To: References: Message-ID: <2ED3AD49-1E9F-4BCB-BE2F-D722C59A57E3@us.es> Hi Matt, thank you for your response! The most likely thing is that the nullspace is slightly wrong, or the convergence test is not accounting for the nullspace somehow. However, since so many equations are inactive, I would consider projecting the problem. I might either 1) Select a subset of the mesh to phrase the problem on. In you use DMPlex, you could use DMPlexFilter(). I?am using a DMSwarm mesh so I can?t use this function. 2) Use a constraint to get a subsystem for Newton using SNESVI. The SNESVIRS will project the linear system onto a subspace without the constraints. I assume you meant SNESVINEWTONRSLS (former SNESVI). I?ve been digging on this approach. However,I don?t think this fits my problem since I can not define the list of active nodes using upper and lower limits like in: petsc.org [X] Thanks, Miguel -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon May 5 14:43:39 2025 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 5 May 2025 15:43:39 -0400 Subject: [petsc-users] Problems with the null-space removal In-Reply-To: <2ED3AD49-1E9F-4BCB-BE2F-D722C59A57E3@us.es> References: <2ED3AD49-1E9F-4BCB-BE2F-D722C59A57E3@us.es> Message-ID: On Mon, May 5, 2025 at 2:08?PM MIGUEL MOLINOS PEREZ wrote: > Hi Matt, thank you for your response! > > The most likely thing is that the nullspace is slightly wrong, or the > convergence test is not accounting for the nullspace somehow. > > However, since so many equations are inactive, I would consider projecting > the problem. I might either > > 1) Select a subset of the mesh to phrase the problem on. In you use > DMPlex, you could use DMPlexFilter(). > > > I?am using a DMSwarm mesh so I can?t use this function. > Oh, so you want to turn off some Swarm nodes? Joe has a DMSwarmCreatreSubswarm() function somewhere. This might work. > 2) Use a constraint to get a subsystem for Newton using SNESVI. The > SNESVIRS will project the linear system onto a subspace without the > constraints. > > > I assume you meant *SNESVINEWTONRSLS (former SNESVI)*. > This is a subtype of SNESVI as it is a particular implementation. > I?ve been digging on this approach. However,I don?t think this fits my > problem since I can not define the list of active nodes using upper and > lower limits like in: > The way I am thinking, you must have some test to determine what nodes are active. I was supposing that test depending on the solution at those nodes. If that is true, you can use that test as the constraint. Thanks, Matt > > petsc.org > > > > > Thanks, > Miguel > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ezvVIFtMgyEOMIRv8YEwgp7LXjHr2RxGixxJ5mO6iKk1tUEbvE5XyAUc3DcQzPkbbby5AcQBnFG79ZOkRSWY$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Tue May 6 02:03:36 2025 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Tue, 6 May 2025 07:03:36 +0000 Subject: [petsc-users] Problems with the null-space removal In-Reply-To: References: <2ED3AD49-1E9F-4BCB-BE2F-D722C59A57E3@us.es> Message-ID: <44753CA7-F89B-480B-9A61-343CAA33DA00@us.es> Oh, so you want to turn off some Swarm nodes? Joe has a DMSwarmCreatreSubswarm() function somewhere. This might work. This would be awesome! Thanks, Miguel On 5 May 2025, at 21:43, Matthew Knepley wrote: On Mon, May 5, 2025 at 2:08?PM MIGUEL MOLINOS PEREZ > wrote: Hi Matt, thank you for your response! The most likely thing is that the nullspace is slightly wrong, or the convergence test is not accounting for the nullspace somehow. However, since so many equations are inactive, I would consider projecting the problem. I might either 1) Select a subset of the mesh to phrase the problem on. In you use DMPlex, you could use DMPlexFilter(). I?am using a DMSwarm mesh so I can?t use this function. Oh, so you want to turn off some Swarm nodes? Joe has a DMSwarmCreatreSubswarm() function somewhere. This might work. 2) Use a constraint to get a subsystem for Newton using SNESVI. The SNESVIRS will project the linear system onto a subspace without the constraints. I assume you meant SNESVINEWTONRSLS (former SNESVI). This is a subtype of SNESVI as it is a particular implementation. I?ve been digging on this approach. However,I don?t think this fits my problem since I can not define the list of active nodes using upper and lower limits like in: The way I am thinking, you must have some test to determine what nodes are active. I was supposing that test depending on the solution at those nodes. If that is true, you can use that test as the constraint. Thanks, Matt petsc.org [X] Thanks, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!f6Wri2FOVoxR4vNWPeTNooN5UqXpPFljoLF1knQliadLxT1WUzm9wbzGQxQObpniv3dLft6oFLHdBfNUjS4jDw$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jp.salazar at pm.me Wed May 7 23:26:00 2025 From: jp.salazar at pm.me (Juan Salazar) Date: Thu, 08 May 2025 04:26:00 +0000 Subject: [petsc-users] dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib Message-ID: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> Dear PETSc users, I am probably doing something dumb, but since I recently updated macOS, I am getting the following errors when running `make check`: dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib . I am also not able to compile codes that previously worked with petsc because of the same issues. The dylib is not linked, even tough it is in the path that apparently is searched. Below is my configure command ./configure --with-debugging=0 --download-cmake --download-hypre --download-parmetis --download-metis --download-ptscotch --download-mumps --download-scalapack --with-precision=double --with-shared-libraries=1 --with-scalar-type=real --with-fc=mpifort --with-cc=mpicc --with-cxx=mpicxx CXX_LINKER_FLAGS=-Wl,-rpath,/opt/homebrew/lib CFLAGS="-g -O2 -fPIC" CXXFLAGS="-g -O2 -fPIC" FCFLAGS="-g -O2" FFLAGS="-g -O2" LDFLAGS="-Wl,-ld_classic -Wl,-commons,use_dylibs" MAKEFLAGS=w --download-bison Attached are the configure.log and make.log and check.log files. ? petsc-3.23.1 % mpifort --version GNU Fortran (Homebrew GCC 14.2.0_1) 14.2.0 Copyright (C) 2024 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ? petsc-3.23.1 % mpicc --version Apple clang version 17.0.0 (clang-1700.0.13.3) Target: arm64-apple-darwin24.4.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin ? petsc-3.23.1 % mpicxx --version Apple clang version 17.0.0 (clang-1700.0.13.3) Target: arm64-apple-darwin24.4.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin macOS Sequoia 15.4.1 (24E263) Any help is much appreciated, Juan S. -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: application/octet-stream Size: 122134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: check.log Type: application/octet-stream Size: 30654 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 4881958 bytes Desc: not available URL: From andrsd at gmail.com Thu May 8 07:38:15 2025 From: andrsd at gmail.com (David Andrs) Date: Thu, 8 May 2025 06:38:15 -0600 Subject: [petsc-users] dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib In-Reply-To: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> References: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> Message-ID: Hi Juan, I recently bumped into a similar issue (not with PETSc). It is a general issue with the newest macOS where a duplicate rpath prevents you from running the binary. The info is buried in the error message when you run the binary. I saw people "fixing" the issue by downgrading Xcode to 16.2 - if my memory serves well. Also you can inspect your binary via `otool -l` to check for the duplicate entries and remove them via install_name_tool (not a solution, but a diagnosing step). So, maybe check if this is the problem you are running into. Side note: conda environment has the same problem and users are advised to update to the latest ld64 (from conda-forge). With regards, -- David On Wed, May 7, 2025 at 10:27?PM Juan Salazar via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear PETSc users, > > I am probably doing something dumb, but since I recently updated macOS, I > am getting the following errors when running `make check`: dyld[19098]: > Library not loaded: @rpath/libHYPRE-2.32.0.dylib . I am also not able to > compile codes that previously worked with petsc because of the same issues. > The dylib is not linked, even tough it is in the path that apparently is > searched. > > Below is my configure command > > ./configure --with-debugging=0 --download-cmake --download-hypre > --download-parmetis --download-metis --download-ptscotch --download-mumps > --download-scalapack --with-precision=double --with-shared-libraries=1 > --with-scalar-type=real --with-fc=mpifort --with-cc=mpicc --with-cxx=mpicxx > CXX_LINKER_FLAGS=-Wl,-rpath,/opt/homebrew/lib CFLAGS="-g -O2 -fPIC" > CXXFLAGS="-g -O2 -fPIC" FCFLAGS="-g -O2" FFLAGS="-g -O2" > LDFLAGS="-Wl,-ld_classic -Wl,-commons,use_dylibs" MAKEFLAGS=w > --download-bison > > Attached are the configure.log and make.log and check.log files. > > ? petsc-3.23.1 % mpifort --version > GNU Fortran (Homebrew GCC 14.2.0_1) 14.2.0 > Copyright (C) 2024 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > ? petsc-3.23.1 % mpicc --version > Apple clang version 17.0.0 (clang-1700.0.13.3) > Target: arm64-apple-darwin24.4.0 > Thread model: posix > InstalledDir: > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin > > ? petsc-3.23.1 % mpicxx --version > Apple clang version 17.0.0 (clang-1700.0.13.3) > Target: arm64-apple-darwin24.4.0 > Thread model: posix > InstalledDir: > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin > > macOS Sequoia 15.4.1 (24E263) > > Any help is much appreciated, > Juan S. > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu May 8 07:44:22 2025 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 8 May 2025 08:44:22 -0400 Subject: [petsc-users] dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib In-Reply-To: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> References: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> Message-ID: On Thu, May 8, 2025 at 12:27?AM Juan Salazar via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear PETSc users, > There are some weird path things going on, When you configure, you are in PWD=/Users/jsalazar/openfoam/jsalazar-9/ThirdParty/petsc-3.23.1 but when it is looking for dynamic libraries, it gets Reason: tried: '/Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1/arch-darwin-c-opt/lib/libHYPRE-2.32.0.dylib' (duplicate LC_RPATH '/opt/homebrew/Cellar/open-mpi/5.0.7/lib'), First, never ever ever ever used mixed case directories on MacOS. It has the worst of all possible systems. It is insensitive for some checks, but has case sensitive storage, so other checks especially in programs fail. Second there is some mismatch here between /Volumes and /Users/jsalazar. I recommend reinstalling completely in the location you want to be in. Thanks, Matt > I am probably doing something dumb, but since I recently updated macOS, I > am getting the following errors when running `make check`: dyld[19098]: > Library not loaded: @rpath/libHYPRE-2.32.0.dylib . I am also not able to > compile codes that previously worked with petsc because of the same issues. > The dylib is not linked, even tough it is in the path that apparently is > searched. > > Below is my configure command > > ./configure --with-debugging=0 --download-cmake --download-hypre > --download-parmetis --download-metis --download-ptscotch --download-mumps > --download-scalapack --with-precision=double --with-shared-libraries=1 > --with-scalar-type=real --with-fc=mpifort --with-cc=mpicc --with-cxx=mpicxx > CXX_LINKER_FLAGS=-Wl,-rpath,/opt/homebrew/lib CFLAGS="-g -O2 -fPIC" > CXXFLAGS="-g -O2 -fPIC" FCFLAGS="-g -O2" FFLAGS="-g -O2" > LDFLAGS="-Wl,-ld_classic -Wl,-commons,use_dylibs" MAKEFLAGS=w > --download-bison > > Attached are the configure.log and make.log and check.log files. > > ? petsc-3.23.1 % mpifort --version > GNU Fortran (Homebrew GCC 14.2.0_1) 14.2.0 > Copyright (C) 2024 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > ? petsc-3.23.1 % mpicc --version > Apple clang version 17.0.0 (clang-1700.0.13.3) > Target: arm64-apple-darwin24.4.0 > Thread model: posix > InstalledDir: > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin > > ? petsc-3.23.1 % mpicxx --version > Apple clang version 17.0.0 (clang-1700.0.13.3) > Target: arm64-apple-darwin24.4.0 > Thread model: posix > InstalledDir: > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin > > macOS Sequoia 15.4.1 (24E263) > > Any help is much appreciated, > Juan S. > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bDJPz2DPLd8evhN5qs-uCttpE8YBYq3IextkJAnaY_4KWG8-LAVyzt_OwBEE1g5SULf4v8Jy5YvAM3iKeunA$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu May 8 08:57:29 2025 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 8 May 2025 09:57:29 -0400 Subject: [petsc-users] dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib In-Reply-To: References: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> Message-ID: On Thu, May 8, 2025 at 9:15?AM David Andrs wrote: > Hi Juan, > > I recently bumped into a similar issue (not with PETSc). It is a general > issue with the newest macOS where a duplicate rpath prevents you from > running the binary. The info is buried in the error message when you run > the binary. I saw people "fixing" the issue by downgrading Xcode to 16.2 - > if my memory serves well. Also you can inspect your binary via `otool -l` > to check for the duplicate entries and remove them via install_name_tool > (not a solution, but a diagnosing step). So, maybe check if this is the > problem you are running into. > Thanks! I will have to remember this one. Apple keeps finding ways to make us less productive. Matt > Side note: conda environment has the same problem and users are advised to > update to the latest ld64 (from conda-forge). > > With regards, > -- > David > > > On Wed, May 7, 2025 at 10:27?PM Juan Salazar via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Dear PETSc users, >> >> I am probably doing something dumb, but since I recently updated macOS, I >> am getting the following errors when running `make check`: dyld[19098]: >> Library not loaded: @rpath/libHYPRE-2.32.0.dylib . I am also not able to >> compile codes that previously worked with petsc because of the same issues. >> The dylib is not linked, even tough it is in the path that apparently is >> searched. >> >> Below is my configure command >> >> ./configure --with-debugging=0 --download-cmake --download-hypre >> --download-parmetis --download-metis --download-ptscotch --download-mumps >> --download-scalapack --with-precision=double --with-shared-libraries=1 >> --with-scalar-type=real --with-fc=mpifort --with-cc=mpicc --with-cxx=mpicxx >> CXX_LINKER_FLAGS=-Wl,-rpath,/opt/homebrew/lib CFLAGS="-g -O2 -fPIC" >> CXXFLAGS="-g -O2 -fPIC" FCFLAGS="-g -O2" FFLAGS="-g -O2" >> LDFLAGS="-Wl,-ld_classic -Wl,-commons,use_dylibs" MAKEFLAGS=w >> --download-bison >> >> Attached are the configure.log and make.log and check.log files. >> >> ? petsc-3.23.1 % mpifort --version >> GNU Fortran (Homebrew GCC 14.2.0_1) 14.2.0 >> Copyright (C) 2024 Free Software Foundation, Inc. >> This is free software; see the source for copying conditions. There is NO >> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >> PURPOSE. >> >> ? petsc-3.23.1 % mpicc --version >> Apple clang version 17.0.0 (clang-1700.0.13.3) >> Target: arm64-apple-darwin24.4.0 >> Thread model: posix >> InstalledDir: >> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin >> >> ? petsc-3.23.1 % mpicxx --version >> Apple clang version 17.0.0 (clang-1700.0.13.3) >> Target: arm64-apple-darwin24.4.0 >> Thread model: posix >> InstalledDir: >> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin >> >> macOS Sequoia 15.4.1 (24E263) >> >> Any help is much appreciated, >> Juan S. >> >> >> >> >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!foYFpXpWd_2fpOCFImBaX5XJCne4IvL_yUutmn2DFXcM7OgZ5IwCQNDVbRbCQBgFuwkIs1sYAKeY7pD4vxvO$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Thu May 8 09:22:37 2025 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Thu, 8 May 2025 14:22:37 +0000 Subject: [petsc-users] Problems with the null-space removal In-Reply-To: <44753CA7-F89B-480B-9A61-343CAA33DA00@us.es> References: <2ED3AD49-1E9F-4BCB-BE2F-D722C59A57E3@us.es> <44753CA7-F89B-480B-9A61-343CAA33DA00@us.es> Message-ID: This function ? DMSwarmCreatreSubswarm? is published somewhere? I can?t find it in the main branch of PETSc. Thanks, Miguel On May 6, 2025, at 9:03?AM, MIGUEL MOLINOS PEREZ wrote: Oh, so you want to turn off some Swarm nodes? Joe has a DMSwarmCreatreSubswarm() function somewhere. This might work. This would be awesome! Thanks, Miguel On 5 May 2025, at 21:43, Matthew Knepley wrote: On Mon, May 5, 2025 at 2:08?PM MIGUEL MOLINOS PEREZ > wrote: Hi Matt, thank you for your response! The most likely thing is that the nullspace is slightly wrong, or the convergence test is not accounting for the nullspace somehow. However, since so many equations are inactive, I would consider projecting the problem. I might either 1) Select a subset of the mesh to phrase the problem on. In you use DMPlex, you could use DMPlexFilter(). I?am using a DMSwarm mesh so I can?t use this function. Oh, so you want to turn off some Swarm nodes? Joe has a DMSwarmCreatreSubswarm() function somewhere. This might work. 2) Use a constraint to get a subsystem for Newton using SNESVI. The SNESVIRS will project the linear system onto a subspace without the constraints. I assume you meant SNESVINEWTONRSLS (former SNESVI). This is a subtype of SNESVI as it is a particular implementation. I?ve been digging on this approach. However,I don?t think this fits my problem since I can not define the list of active nodes using upper and lower limits like in: The way I am thinking, you must have some test to determine what nodes are active. I was supposing that test depending on the solution at those nodes. If that is true, you can use that test as the constraint. Thanks, Matt petsc.org [X] Thanks, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eu4LXQ_gZoOkWMB06MuZCy2wteWtP_sw5qGUeEZ-JyK16VZ-D_QrioHDxE6a1wl6HWP27k6EVFqpYchd8cClSQ$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephpu at buffalo.edu Thu May 8 11:43:23 2025 From: josephpu at buffalo.edu (Joseph Pusztay) Date: Thu, 8 May 2025 16:43:23 +0000 Subject: [petsc-users] Problems with the null-space removal In-Reply-To: References: <2ED3AD49-1E9F-4BCB-BE2F-D722C59A57E3@us.es> <44753CA7-F89B-480B-9A61-343CAA33DA00@us.es> Message-ID: Hi Miguel, I believe what you may be looking for is DMSwarmGetCellSwarm which extracts the fields from the swarm for a specified cell https://urldefense.us/v3/__https://petsc.org/release/manualpages/DMSwarm/DMSwarmGetCellSwarm/__;!!G_uCfscf7eWS!ZNzXAukaJWoy9gfF0ZpkipSh7IvFVWu9UsybjhiA8z5_GHxWHwJ60oFlfUGwwh73aGqUO3XB4W8Mo4B5m7I4YoqhM1c$ Best, Joe ________________________________ From: MIGUEL MOLINOS PEREZ Sent: Thursday, May 8, 2025 10:22 AM To: Matthew Knepley Cc: petsc-users at mcs.anl.gov ; Joseph Pusztay Subject: Re: [petsc-users] Problems with the null-space removal You don't often get email from mmolinos at us.es. Learn why this is important This function ? DMSwarmCreatreSubswarm? is published somewhere? I can?t find it in the main branch of PETSc. Thanks, Miguel On May 6, 2025, at 9:03?AM, MIGUEL MOLINOS PEREZ wrote: Oh, so you want to turn off some Swarm nodes? Joe has a DMSwarmCreatreSubswarm() function somewhere. This might work. This would be awesome! Thanks, Miguel On 5 May 2025, at 21:43, Matthew Knepley wrote: On Mon, May 5, 2025 at 2:08?PM MIGUEL MOLINOS PEREZ > wrote: Hi Matt, thank you for your response! The most likely thing is that the nullspace is slightly wrong, or the convergence test is not accounting for the nullspace somehow. However, since so many equations are inactive, I would consider projecting the problem. I might either 1) Select a subset of the mesh to phrase the problem on. In you use DMPlex, you could use DMPlexFilter(). I?am using a DMSwarm mesh so I can?t use this function. Oh, so you want to turn off some Swarm nodes? Joe has a DMSwarmCreatreSubswarm() function somewhere. This might work. 2) Use a constraint to get a subsystem for Newton using SNESVI. The SNESVIRS will project the linear system onto a subspace without the constraints. I assume you meant SNESVINEWTONRSLS (former SNESVI). This is a subtype of SNESVI as it is a particular implementation. I?ve been digging on this approach. However,I don?t think this fits my problem since I can not define the list of active nodes using upper and lower limits like in: The way I am thinking, you must have some test to determine what nodes are active. I was supposing that test depending on the solution at those nodes. If that is true, you can use that test as the constraint. Thanks, Matt petsc.org [X] Thanks, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZNzXAukaJWoy9gfF0ZpkipSh7IvFVWu9UsybjhiA8z5_GHxWHwJ60oFlfUGwwh73aGqUO3XB4W8Mo4B5m7I4tiffvTQ$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmolinos at us.es Thu May 8 13:13:20 2025 From: mmolinos at us.es (MIGUEL MOLINOS PEREZ) Date: Thu, 8 May 2025 18:13:20 +0000 Subject: [petsc-users] Problems with the null-space removal In-Reply-To: References: <2ED3AD49-1E9F-4BCB-BE2F-D722C59A57E3@us.es> <44753CA7-F89B-480B-9A61-343CAA33DA00@us.es> Message-ID: <76632893-F567-44B1-8E2C-833794B55EE3@us.es> Hi Joe I don?t think this function will work for me since it extract particles cell-wise. However, I think I can use this function as basis to build what I have in mind? Thanks, Miguel On May 8, 2025, at 6:43?PM, Joseph Pusztay wrote: Hi Miguel, I believe what you may be looking for is DMSwarmGetCellSwarm which extracts the fields from the swarm for a specified cell https://urldefense.us/v3/__https://petsc.org/release/manualpages/DMSwarm/DMSwarmGetCellSwarm/__;!!G_uCfscf7eWS!cwnn3sRq7di9mEjZz_VPpuzbpto02sSR7pko8ndG7obAmHU1vHXnI8cShm67H2Js9wkWYo1ieIXb4e6qjjIOng$ Best, Joe ________________________________ From: MIGUEL MOLINOS PEREZ > Sent: Thursday, May 8, 2025 10:22 AM To: Matthew Knepley > Cc: petsc-users at mcs.anl.gov >; Joseph Pusztay > Subject: Re: [petsc-users] Problems with the null-space removal You don't often get email from mmolinos at us.es. Learn why this is important This function ? DMSwarmCreatreSubswarm? is published somewhere? I can?t find it in the main branch of PETSc. Thanks, Miguel On May 6, 2025, at 9:03?AM, MIGUEL MOLINOS PEREZ wrote: Oh, so you want to turn off some Swarm nodes? Joe has a DMSwarmCreatreSubswarm() function somewhere. This might work. This would be awesome! Thanks, Miguel On 5 May 2025, at 21:43, Matthew Knepley wrote: On Mon, May 5, 2025 at 2:08?PM MIGUEL MOLINOS PEREZ > wrote: Hi Matt, thank you for your response! The most likely thing is that the nullspace is slightly wrong, or the convergence test is not accounting for the nullspace somehow. However, since so many equations are inactive, I would consider projecting the problem. I might either 1) Select a subset of the mesh to phrase the problem on. In you use DMPlex, you could use DMPlexFilter(). I?am using a DMSwarm mesh so I can?t use this function. Oh, so you want to turn off some Swarm nodes? Joe has a DMSwarmCreatreSubswarm() function somewhere. This might work. 2) Use a constraint to get a subsystem for Newton using SNESVI. The SNESVIRS will project the linear system onto a subspace without the constraints. I assume you meant SNESVINEWTONRSLS (former SNESVI). This is a subtype of SNESVI as it is a particular implementation. I?ve been digging on this approach. However,I don?t think this fits my problem since I can not define the list of active nodes using upper and lower limits like in: The way I am thinking, you must have some test to determine what nodes are active. I was supposing that test depending on the solution at those nodes. If that is true, you can use that test as the constraint. Thanks, Matt petsc.org [X] Thanks, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cwnn3sRq7di9mEjZz_VPpuzbpto02sSR7pko8ndG7obAmHU1vHXnI8cShm67H2Js9wkWYo1ieIXb4e5B3deHRA$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu May 8 20:31:15 2025 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 8 May 2025 21:31:15 -0400 Subject: [petsc-users] dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib In-Reply-To: <12288579-43BD-4C8D-AC56-84AE15B0000D@pm.me> References: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> <12288579-43BD-4C8D-AC56-84AE15B0000D@pm.me> Message-ID: On Thu, May 8, 2025 at 6:22?PM Juan Salazar wrote: > Thank you for the feedback Matthew, > > I should explain why there is a difference in the path variables. I am > compiling PETSc on case-sensitive Apple File System as a volume mounted at > /Volume/OpenFOAM. This is because I compile OpenFOAM natively (it is > developed for Linux, but with some tweaks it can be compiled on macOS). I > use the PETSc library within custom OpenFOAM applications (solvers). > Therefore I install PETSc in the same volume. > I believe it is as David said. The dynamic linker cannot load it due to multiple paths. You should be able to see that with otool -L on the executable. Did you see his message? Thanks, Matt Reason: tried: '/Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1/arch-darwin-c-opt/lib/libHYPRE-2.32.0.dylib' (duplicate LC_RPATH '/opt/homebrew/Cellar/open-mpi/5.0.7/lib'), > Depending on how I reach the installation folder, the PWD variable will be > set differently. > > Last login: Thu May 8 19:05:51 on ttys004 > ? ~ % cd /Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 > ? petsc-3.23.1 % pwd > /Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 > > Last login: Thu May 8 19:07:38 on ttys004 > ? ~ % cd ~/openfoam/jsalazar-9/ThirdParty/petsc-3.23.1 > ? petsc-3.23.1 % pwd > /Users/jsalazar/openfoam/jsalazar-9/ThirdParty/petsc-3.23.1 > > I noticed that within the python configure script the path is obtained > invariably through the command os.getcwd(). > > ? petsc-3.23.1 % python > Python 3.12.10 (main, Apr 8 2025, 11:35:47) [Clang 16.0.0 > (clang-1600.0.26.6)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import os > >>> os.getcwd() > '/Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1' > > This result is the same independent of how I got to the folder. > > So I ran ./configure again making sure that PWD is the same as > os.getcwd(). Unfortunately this did not solve the problem. I believe the > paths are consistent now. > > Thanks again, > Juan S. > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b44h-sLwTGYsZVJkeoCYZpHibxlRGYKesJNr1WEgrw33x5IuPBSXOdcY-D0WcN3UZI9-iV226YQy0HicD5FV$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hardik.kothari at corintis.com Fri May 9 08:54:34 2025 From: hardik.kothari at corintis.com (Hardik Kothari) Date: Fri, 9 May 2025 13:54:34 +0000 Subject: [petsc-users] Solving Stokes problem in high aspect ratio domains References: <80f2df75-0faf-40ef-9409-a0422cfa7303.d0d10b7c-e93e-4ca5-8394-d4339a327c55.db30f155-f69a-4d1b-8a4d-6dafd7602ac1@emailsignatures365.codetwo.com> Message-ID: <4B3A1EE5-1A34-42B2-B36D-E89436F7EB25@corintis.com> Dear PETSc team, We are solving the Stokes equations using PETSc (via Firedrake) on a highly anisotropic 3D domain (L_x=1, L_y=0.01, L_z=0.1). In this setup, standard Schur complement preconditioners using a mass inverse for pressure struggle to converge. We could solve the problem with the LSC preconditioner (solver parameters are shown in the script). We have the following questions: * Why standard preconditioners struggle in such domains? * Why is the preconditioned residual norm for the Schur complement system much higher than the true residual norm? * Would you recommend alternative or more robust preconditioners for such geometries? Thank you for your help. Best regards, Hardik HARDIK KOTHARI hardik.kothari at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne [https://urldefense.us/v3/__https://storcor.s3.eu-central-1.amazonaws.com/logos/Logo-black.png__;!!G_uCfscf7eWS!coZIkzX5gHWdyGx54LSoh7iP6KZ7AV-0Oa7FTFrMVAzIYrBCsVJ0vqY0hLvcIqcrdQtxcmcdpr25Hhsper7CRJwNrHrjfDsq$ ] Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: slimStokes.py Type: text/x-python-script Size: 3815 bytes Desc: slimStokes.py URL: From jp.salazar at pm.me Thu May 8 17:22:04 2025 From: jp.salazar at pm.me (Juan Salazar) Date: Thu, 08 May 2025 22:22:04 +0000 Subject: [petsc-users] dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib In-Reply-To: References: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> Message-ID: <12288579-43BD-4C8D-AC56-84AE15B0000D@pm.me> Thank you for the feedback Matthew, I should explain why there is a difference in the path variables. I am compiling PETSc on case-sensitive Apple File System as a volume mounted at /Volume/OpenFOAM. This is because I compile OpenFOAM natively (it is developed for Linux, but with some tweaks it can be compiled on macOS). I use the PETSc library within custom OpenFOAM applications (solvers). Therefore I install PETSc in the same volume. Depending on how I reach the installation folder, the PWD variable will be set differently. Last login: Thu May 8 19:05:51 on ttys004 ? ~ % cd /Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 ? petsc-3.23.1 % pwd /Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 Last login: Thu May 8 19:07:38 on ttys004 ? ~ % cd ~/openfoam/jsalazar-9/ThirdParty/petsc-3.23.1 ? petsc-3.23.1 % pwd /Users/jsalazar/openfoam/jsalazar-9/ThirdParty/petsc-3.23.1 I noticed that within the python configure script the path is obtained invariably through the command os.getcwd(). ? petsc-3.23.1 % python Python 3.12.10 (main, Apr 8 2025, 11:35:47) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.getcwd() '/Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1' This result is the same independent of how I got to the folder. So I ran ./configure again making sure that PWD is the same as os.getcwd(). Unfortunately this did not solve the problem. I believe the paths are consistent now. Thanks again, Juan S. -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: application/octet-stream Size: 122715 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: check.log Type: application/octet-stream Size: 30605 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 11346418 bytes Desc: not available URL: From pierre at joliv.et Fri May 9 10:34:14 2025 From: pierre at joliv.et (Pierre Jolivet) Date: Fri, 9 May 2025 17:34:14 +0200 Subject: [petsc-users] dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib In-Reply-To: <12288579-43BD-4C8D-AC56-84AE15B0000D@pm.me> References: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> <12288579-43BD-4C8D-AC56-84AE15B0000D@pm.me> Message-ID: <61B93977-1FD5-4D10-96C3-40661BBED65E@joliv.et> Did you install Open MPI via Homebrew? I don?t think this will work (at least, I have yet to see a single person make it work with the recent changes with Apple linker). I would: 1) remove all this unnecessary stuff CXX_LINKER_FLAGS=-Wl,-rpath,/opt/homebrew/lib CFLAGS="-g -O2 -fPIC" CXXFLAGS="-g -O2 -fPIC" FCFLAGS="-g -O2" FFLAGS="-g -O2" LDFLAGS="-Wl,-ld_classic -Wl,-commons,use_dylibs" MAKEFLAGS=w These are either: default parameters or low-level parameters that are likely to break stuff if not set appropriately. 2) add something like --download-mpich or --download-openmpi to your configure options 3) start fresh by removing your previous PETSC_ARCH directory Thanks, Pierre > On 9 May 2025, at 12:22?AM, Juan Salazar via petsc-users wrote: > > Thank you for the feedback Matthew, > > I should explain why there is a difference in the path variables. I am compiling PETSc on case-sensitive Apple File System as a volume mounted at /Volume/OpenFOAM. This is because I compile OpenFOAM natively (it is developed for Linux, but with some tweaks it can be compiled on macOS). I use the PETSc library within custom OpenFOAM applications (solvers). Therefore I install PETSc in the same volume. > > Depending on how I reach the installation folder, the PWD variable will be set differently. > > Last login: Thu May 8 19:05:51 on ttys004 > ? ~ % cd /Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 > ? petsc-3.23.1 % pwd > /Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 > > Last login: Thu May 8 19:07:38 on ttys004 > ? ~ % cd ~/openfoam/jsalazar-9/ThirdParty/petsc-3.23.1 > ? petsc-3.23.1 % pwd > /Users/jsalazar/openfoam/jsalazar-9/ThirdParty/petsc-3.23.1 > > I noticed that within the python configure script the path is obtained invariably through the command os.getcwd(). > > ? petsc-3.23.1 % python > Python 3.12.10 (main, Apr 8 2025, 11:35:47) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin > Type "help", "copyright", "credits" or "license" for more information. >>>> import os >>>> os.getcwd() > '/Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1' > > This result is the same independent of how I got to the folder. > > So I ran ./configure again making sure that PWD is the same as os.getcwd(). Unfortunately this did not solve the problem. I believe the paths are consistent now. > > Thanks again, > Juan S. > > > > > > > From balay.anl at fastmail.org Fri May 9 10:37:31 2025 From: balay.anl at fastmail.org (Satish Balay) Date: Fri, 9 May 2025 10:37:31 -0500 (CDT) Subject: [petsc-users] dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib In-Reply-To: References: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> <12288579-43BD-4C8D-AC56-84AE15B0000D@pm.me> Message-ID: <1aeffdd3-6a50-c448-905a-4d59ce8d888a@fastmail.org> >>> Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --with-debugging=0 --download-make --download-cmake --download-hypre --download-parmetis --download-metis --download-ptscotch --download-mumps --download-scalapack --with-precision=double --with-shared-libraries=1 --with-scalar-type=real --with-fc=mpifort --with-cc=mpicc --with-cxx=mpicxx CXX_LINKER_FLAGS=-Wl,-rpath,/opt/homebrew/lib CFLAGS="-g -O2 -fPIC" CXXFLAGS="-g -O2 -fPIC" FCFLAGS="-g -O2" FFLAGS="-g -O2" LDFLAGS="-Wl,-ld_classic -Wl,-commons,use_dylibs" MAKEFLAGS=w --download-bison <<< Hm LDFLAGS="-Wl,-ld_classic -Wl,-commons,use_dylibs" should be needed anymore - well atleast with latest --download-mpich. >>> Executing: mpicc -show stdout: clang -I/opt/homebrew/Cellar/open-mpi/5.0.7/include -L/opt/homebrew/Cellar/open-mpi/5.0.7/lib -lmpi Executing: mpicc --version stdout: Apple clang version 17.0.0 (clang-1700.0.13.3) Target: arm64-apple-darwin24.4.0 <<<< We use the same xcode/clang version for our CI - and this issue doesn't come up here. >>> balay at npro ~ % clang --version Apple clang version 17.0.0 (clang-1700.0.13.3) Target: arm64-apple-darwin24.4.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin <<< Ok - just tried: >>> balay at npro petsc % ./configure --with-debugging=0 --download-make --download-cmake --download-hypre --download-parmetis --download-metis --download-ptscotch --download-mumps --download-scalapack --with-precision=double --with-shared-libraries=1 --with-scalar-type=real --download-mpich CC=clang CXX=clang++ FC=gfortran COPTFLAGS="-g -O2 -fPIC" CXXFOPTLAGS="-g -O2 -fPIC" FOPTFLAGS="-g -O2" --download-bison xxx=======================================================================================xxx Configure stage complete. Now build PETSc libraries with: make PETSC_DIR=/Users/balay/petsc PETSC_ARCH=arch-darwin-c-opt all xxx=======================================================================================xxx balay at npro petsc % make && make check FC arch-darwin-c-opt/obj/src/snes/ftn-mod/petscsnesmod.o FC arch-darwin-c-opt/obj/src/ts/ftn-mod/petsctsmod.o FC arch-darwin-c-opt/obj/src/tao/ftn-mod/petsctaomod.o CLINKER arch-darwin-c-opt/lib/libpetsc.3.23.1.dylib ld: warning: reducing alignment of section __DATA,__common from 0x8000 to 0x4000 because it exceeds segment maximum alignment DSYMUTIL arch-darwin-c-opt/lib/libpetsc.3.23.1.dylib make[3]: Leaving directory '/Users/balay/petsc' make[2]: Leaving directory '/Users/balay/petsc' ========================================= Now to check if the libraries are working do: make PETSC_DIR=/Users/balay/petsc PETSC_ARCH=arch-darwin-c-opt check ========================================= Running PETSc check examples to verify correct installation Using PETSC_DIR=/Users/balay/petsc and PETSC_ARCH=arch-darwin-c-opt C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed PETSc check examples balay at npro petsc % <<<< Would this alternate work for you? Satish On Thu, 8 May 2025, Matthew Knepley wrote: > On Thu, May 8, 2025 at 6:22?PM Juan Salazar wrote: > > > Thank you for the feedback Matthew, > > > > I should explain why there is a difference in the path variables. I am > > compiling PETSc on case-sensitive Apple File System as a volume mounted at > > /Volume/OpenFOAM. This is because I compile OpenFOAM natively (it is > > developed for Linux, but with some tweaks it can be compiled on macOS). I > > use the PETSc library within custom OpenFOAM applications (solvers). > > Therefore I install PETSc in the same volume. > > > > I believe it is as David said. The dynamic linker cannot load it due to > multiple paths. You should be able to see that with otool -L on the > executable. Did you see his message? > > Thanks, > > Matt > > Reason: tried: > '/Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1/arch-darwin-c-opt/lib/libHYPRE-2.32.0.dylib' > (duplicate LC_RPATH '/opt/homebrew/Cellar/open-mpi/5.0.7/lib'), > > > > Depending on how I reach the installation folder, the PWD variable will be > > set differently. > > > > Last login: Thu May 8 19:05:51 on ttys004 > > ? ~ % cd /Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 > > ? petsc-3.23.1 % pwd > > /Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 > > > > Last login: Thu May 8 19:07:38 on ttys004 > > ? ~ % cd ~/openfoam/jsalazar-9/ThirdParty/petsc-3.23.1 > > ? petsc-3.23.1 % pwd > > /Users/jsalazar/openfoam/jsalazar-9/ThirdParty/petsc-3.23.1 > > > > I noticed that within the python configure script the path is obtained > > invariably through the command os.getcwd(). > > > > ? petsc-3.23.1 % python > > Python 3.12.10 (main, Apr 8 2025, 11:35:47) [Clang 16.0.0 > > (clang-1600.0.26.6)] on darwin > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import os > > >>> os.getcwd() > > '/Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1' > > > > This result is the same independent of how I got to the folder. > > > > So I ran ./configure again making sure that PWD is the same as > > os.getcwd(). Unfortunately this did not solve the problem. I believe the > > paths are consistent now. > > > > Thanks again, > > Juan S. > > > > > > > > > > > > > > > > From mfadams at lbl.gov Fri May 9 14:08:17 2025 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 9 May 2025 15:08:17 -0400 Subject: [petsc-users] Solving Stokes problem in high aspect ratio domains In-Reply-To: <4B3A1EE5-1A34-42B2-B36D-E89436F7EB25@corintis.com> References: <80f2df75-0faf-40ef-9409-a0422cfa7303.d0d10b7c-e93e-4ca5-8394-d4339a327c55.db30f155-f69a-4d1b-8a4d-6dafd7602ac1@emailsignatures365.codetwo.com> <4B3A1EE5-1A34-42B2-B36D-E89436F7EB25@corintis.com> Message-ID: Hi Hardik, The domain shape is not critical but the element shapes are. Your 100:1 domain aspect ratio is bad if you have N^3 mesh and thus element aspect ratios of 100:1. If that is the case then you probably want to look at semi-coarsening multigrid. Mark On Fri, May 9, 2025 at 9:55?AM Hardik Kothari wrote: > Dear PETSc team, > > We are solving the Stokes equations using PETSc (via Firedrake) on a > highly anisotropic 3D domain (L_x=1, L_y=0.01, L_z=0.1). > > In this setup, standard Schur complement preconditioners using a mass > inverse for pressure struggle to converge. We could solve the problem with > the LSC preconditioner (solver parameters are shown in the script). > > We have the following questions: > > - Why standard preconditioners struggle in such domains? > - Why is the preconditioned residual norm for the Schur complement > system much higher than the true residual norm? > - Would you recommend alternative or more robust preconditioners for > such geometries? > > > Thank you for your help. > > Best regards, > Hardik > > > > > *HARDIK KOTHARI* > *hardik.kothari at corintis.com* > > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > > > [image: Logo-black.png] > Here at Corintis we care for your privacy. That is why we have taken > appropriate measures to ensure that the data you have provided to us is > always secure > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jp.salazar at pm.me Sat May 10 22:41:35 2025 From: jp.salazar at pm.me (Juan Salazar) Date: Sun, 11 May 2025 03:41:35 +0000 Subject: [petsc-users] dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib In-Reply-To: References: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> Message-ID: Dear David, This indeed looks like the culprit! I verified that the libHYPRE-2.32.0.dylib had many duplicate LC_RPATH instances. Upon removing them, the executable (src/snes/tutorials/ex19) complained that the next linked library also has duplicate LC_RPATH instances. So the issue is related to duplicated links. I modified the configure command and removed LDFLAGs and CXX_LINKER_FLAGs. ./configure --with-debugging=0 --download-make --download-cmake --download-hypre --download-parmetis --download-metis --download-ptscotch --download-mumps --download-scalapack --with-precision=double --with-shared-libraries=1 --with-scalar-type=real --with-fc=mpifort --with-cc=mpicc --with-cxx=mpicxx CFLAGS="-g -O2 -fPIC" CXXFLAGS="-g -O2 -fPIC" FCFLAGS="-g -O2" FFLAGS="-g -O2" MAKEFLAGS=w --download-bison And now everything works! ? petsc-3.23.1 % make PETSC_DIR=/Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 PETSC_ARCH=arch-darwin-c-opt check Running PETSc check examples to verify correct installation Using PETSC_DIR=/Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 and PETSC_ARCH=arch-darwin-c-opt C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed PETSc check examples One thing I noticed is that I have LDFLAGS=-Wl,-ld_classic there are still duplicate LC_RPATH. However, If I remove LDFLAGS and instead have CXX_LINKER_FLAG=-Wl,-ld_classic there are no duplicate RC_PATH. Thank you, I learned something today. Also thanks to Matthew, I have to be more careful about different paths, even if they lead to the same place. Sorry I took a long time to reply. Juan S. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jp.salazar at pm.me Sat May 10 22:48:31 2025 From: jp.salazar at pm.me (Juan Salazar) Date: Sun, 11 May 2025 03:48:31 +0000 Subject: [petsc-users] dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib In-Reply-To: <61B93977-1FD5-4D10-96C3-40661BBED65E@joliv.et> References: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> <12288579-43BD-4C8D-AC56-84AE15B0000D@pm.me> <61B93977-1FD5-4D10-96C3-40661BBED65E@joliv.et> Message-ID: Dear Pierre, Indeed, many of the flags are unnecessary, but I was able to keep most of them. I inherited them from another group of developers that compile for linux. I?m the only one working to run this code natively on macOS. What I noticed is that if I pass LDFLAGS=-Wl,-ld_classic there are still duplicate LC_RPATH. However, with CXX_LINKER_FLAGS=-Wl,-ld_classic I get no duplicate LC_RPATH. I indeed am usingOpenMPI via homebrew. But I was able to keep it. Thanks! Juan S. > On May 9, 2025, at 12:34?PM, Pierre Jolivet wrote: > > Did you install Open MPI via Homebrew? > I don?t think this will work (at least, I have yet to see a single person make it work with the recent changes with Apple linker). > I would: > 1) remove all this unnecessary stuff > CXX_LINKER_FLAGS=-Wl,-rpath,/opt/homebrew/lib CFLAGS="-g -O2 -fPIC" CXXFLAGS="-g -O2 -fPIC" FCFLAGS="-g -O2" FFLAGS="-g -O2" LDFLAGS="-Wl,-ld_classic -Wl,-commons,use_dylibs" MAKEFLAGS=w > These are either: default parameters or low-level parameters that are likely to break stuff if not set appropriately. > 2) add something like --download-mpich or --download-openmpi to your configure options > 3) start fresh by removing your previous PETSC_ARCH directory > > Thanks, > Pierre > >> On 9 May 2025, at 12:22?AM, Juan Salazar via petsc-users wrote: >> >> Thank you for the feedback Matthew, >> >> I should explain why there is a difference in the path variables. I am compiling PETSc on case-sensitive Apple File System as a volume mounted at /Volume/OpenFOAM. This is because I compile OpenFOAM natively (it is developed for Linux, but with some tweaks it can be compiled on macOS). I use the PETSc library within custom OpenFOAM applications (solvers). Therefore I install PETSc in the same volume. >> >> Depending on how I reach the installation folder, the PWD variable will be set differently. >> >> Last login: Thu May 8 19:05:51 on ttys004 >> ? ~ % cd /Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 >> ? petsc-3.23.1 % pwd >> /Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 >> >> Last login: Thu May 8 19:07:38 on ttys004 >> ? ~ % cd ~/openfoam/jsalazar-9/ThirdParty/petsc-3.23.1 >> ? petsc-3.23.1 % pwd >> /Users/jsalazar/openfoam/jsalazar-9/ThirdParty/petsc-3.23.1 >> >> I noticed that within the python configure script the path is obtained invariably through the command os.getcwd(). >> >> ? petsc-3.23.1 % python >> Python 3.12.10 (main, Apr 8 2025, 11:35:47) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import os >>>>> os.getcwd() >> '/Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1' >> >> This result is the same independent of how I got to the folder. >> >> So I ran ./configure again making sure that PWD is the same as os.getcwd(). Unfortunately this did not solve the problem. I believe the paths are consistent now. >> >> Thanks again, >> Juan S. >> >> >> >> >> >> >> > From jp.salazar at pm.me Sat May 10 22:57:32 2025 From: jp.salazar at pm.me (Juan Salazar) Date: Sun, 11 May 2025 03:57:32 +0000 Subject: [petsc-users] dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib In-Reply-To: <1aeffdd3-6a50-c448-905a-4d59ce8d888a@fastmail.org> References: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> <12288579-43BD-4C8D-AC56-84AE15B0000D@pm.me> <1aeffdd3-6a50-c448-905a-4d59ce8d888a@fastmail.org> Message-ID: <61000A9E-44AC-4407-8ADA-47FD200CF7AC@pm.me> Dear Sathish, > >>>> > balay at npro petsc % ./configure --with-debugging=0 --download-make --download-cmake --download-hypre --download-parmetis --download-metis --download-ptscotch --download-mumps --download-scalapack --with-precision=double --with-shared-libraries=1 --with-scalar-type=real --download-mpich CC=clang CXX=clang++ FC=gfortran COPTFLAGS="-g -O2 -fPIC" CXXFOPTLAGS="-g -O2 -fPIC" FOPTFLAGS="-g -O2" --download-bison > > > Would this alternate work for you? > Yes! This will certainly work too. The OpenFOAM code that uses PETSc is compiled with OpenMPI, so that is why I kept OpenMPI. I was also able to use the homebrew version. Initially I missed your ./configure reply. It would have saved me a lot of time had I been more attentive. Thanks, Juan S. From knepley at gmail.com Sun May 11 06:03:43 2025 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 11 May 2025 07:03:43 -0400 Subject: [petsc-users] dyld[19098]: Library not loaded: @rpath/libHYPRE-2.32.0.dylib In-Reply-To: References: <15563DAF-70BD-423E-8E22-51EB78486AD8@pm.me> Message-ID: On Sat, May 10, 2025 at 11:42?PM Juan Salazar via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear David, > > This indeed looks like the culprit! I verified that the > libHYPRE-2.32.0.dylib had many duplicate LC_RPATH instances. Upon removing > them, the executable (src/snes/tutorials/ex19) complained that the next > linked library also has duplicate LC_RPATH instances. > Great! I am glad its working now. Thanks, Matt > So the issue is related to duplicated links. I modified the configure > command and removed LDFLAGs and CXX_LINKER_FLAGs. > > ./configure --with-debugging=0 --download-make --download-cmake > --download-hypre --download-parmetis --download-metis --download-ptscotch > --download-mumps --download-scalapack --with-precision=double > --with-shared-libraries=1 --with-scalar-type=real --with-fc=mpifort > --with-cc=mpicc --with-cxx=mpicxx CFLAGS="-g -O2 -fPIC" CXXFLAGS="-g -O2 > -fPIC" FCFLAGS="-g -O2" FFLAGS="-g -O2" MAKEFLAGS=w --download-bison > > And now everything works! > > ? petsc-3.23.1 % make > PETSC_DIR=/Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 > PETSC_ARCH=arch-darwin-c-opt check > Running PETSc check examples to verify correct installation > Using PETSC_DIR=/Volumes/OpenFOAM/jsalazar-9/ThirdParty/petsc-3.23.1 and > PETSC_ARCH=arch-darwin-c-opt > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE > C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > Completed PETSc check examples > > One thing I noticed is that I have LDFLAGS=-Wl,-ld_classic there are > still duplicate LC_RPATH. However, If I remove LDFLAGS and instead have CXX_LINKER_FLAG=-Wl,-ld_classic > there are no duplicate RC_PATH. > > Thank you, I learned something today. Also thanks to Matthew, I have to be > more careful about different paths, even if they lead to the same place. > Sorry I took a long time to reply. > > Juan S. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bJz9P0WtZwmbsj_ISUSTkmRQhZvZ3lpgt5Dg1vh1MALD7ebfywzz8PMLf_csoQPBex7eU6Q8QJeOYn9LSr00$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hj2000 at mail.ustc.edu.cn Sun May 11 10:38:47 2025 From: hj2000 at mail.ustc.edu.cn (hj2000 at mail.ustc.edu.cn) Date: Sun, 11 May 2025 23:38:47 +0800 (GMT+08:00) Subject: [petsc-users] PETSc using problem Message-ID: <1e11e513.160ff6.196bfffb282.Coremail.hj2000@mail.ustc.edu.cn> Dear Developer: I am a user of PETSc, and I am using SNESLineSearchSetPostCheck to perform some actions when the line search fails. However, it seems that this function is only called when the line search is successful. Why is that? Do I need to do something else? My PETSc version is 3.18.6. Below is my code. auto LineSearchPostCheck = []( SNESLineSearch ls, Vec x, Vec y, Vec w, PetscBool* changed_Y, PetscBool* changed_W, void* ) { SNESLineSearchReason lscnv; SNESLineSearchGetReason( ls, &lscnv ); SNES snes = nullptr; SNESLineSearchGetSNES( ls, &snes ); SNESConvergedReason cnv; SNESGetConvergedReason( snes, &cnv ); double lambda = 0.0; SNESLineSearchGetLambda( ls, &lambda ); PetscPrintf( MPI_COMM_WORLD, "lambda=%lf\n", lambda ); if ( lscnv != SNES_LINESEARCH_SUCCEEDED || lambda < 1e-4 ) { VecWAXPY( w, -0.1, y, x ); SNESLineSearchSetReason( ls, SNES_LINESEARCH_SUCCEEDED ); *changed_W = PETSC_TRUE; } return 0; }; SNESLineSearchSetPostCheck( ls, LineSearchPostCheck, nullptr); -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun May 11 10:50:00 2025 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 11 May 2025 11:50:00 -0400 Subject: [petsc-users] PETSc using problem In-Reply-To: <1e11e513.160ff6.196bfffb282.Coremail.hj2000@mail.ustc.edu.cn> References: <1e11e513.160ff6.196bfffb282.Coremail.hj2000@mail.ustc.edu.cn> Message-ID: On Sun, May 11, 2025 at 11:39?AM hj2000--- via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear Developer: > > I am a user of PETSc, and I am using SNESLineSearchSetPostCheck to > perform some actions when the line search fails. > > However, it seems that this function is only called when the line search > is successful. Why is that? > PostCheck is intended to work on the solution found by the line search. > Do I need to do something else? > Yes. We do not have a specific hook for failure, because that would make it a different line search (one which may not obey the advertised properties). Thus we would want your scheme to be its own line search. Thanks, Matt > My PETSc version is 3.18.6. Below is my code. > > > > auto LineSearchPostCheck = []( SNESLineSearch ls, Vec x, Vec y, > Vec w, PetscBool* changed_Y, > > PetscBool* changed_W, void* ) { > > SNESLineSearchReason lscnv; > > SNESLineSearchGetReason( ls, &lscnv ); > > SNES snes = nullptr; > > SNESLineSearchGetSNES( ls, &snes ); > > SNESConvergedReason cnv; > > SNESGetConvergedReason( snes, &cnv ); > > double lambda = 0.0; > > SNESLineSearchGetLambda( ls, &lambda ); > > PetscPrintf( MPI_COMM_WORLD, "lambda=%lf\n", lambda ); > > if ( lscnv != SNES_LINESEARCH_SUCCEEDED || lambda < 1e-4 ) > > { > > VecWAXPY( w, -0.1, y, x ); > > SNESLineSearchSetReason( ls, SNES_LINESEARCH_SUCCEEDED ); > > *changed_W = PETSC_TRUE; > > } > > return 0; > > }; > > SNESLineSearchSetPostCheck( ls, LineSearchPostCheck, nullptr); > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Y6XIK823eTx-iRg60ThDREJsmpqY4jNxIxr-y3_Li6lAA2rs8JvxNjeeUrgfpgbcSCSaRzhBaYiBHw7nGypK$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Sun May 11 13:44:53 2025 From: pierre at joliv.et (Pierre Jolivet) Date: Sun, 11 May 2025 20:44:53 +0200 Subject: [petsc-users] Solving Stokes problem in high aspect ratio domains In-Reply-To: References: <80f2df75-0faf-40ef-9409-a0422cfa7303.d0d10b7c-e93e-4ca5-8394-d4339a327c55.db30f155-f69a-4d1b-8a4d-6dafd7602ac1@emailsignatures365.codetwo.com> <4B3A1EE5-1A34-42B2-B36D-E89436F7EB25@corintis.com> Message-ID: <5BF44380-C0B1-464B-BD56-F5000534EE52@joliv.et> Do you want to refine the geometry or are you fine with the current one? What kind of hardware are you planning on using (GPU, single-node?)? Do you have a configuration for which LSC fails or does not give you good-enough performance? Thanks, Pierre > On 9 May 2025, at 9:08?PM, Mark Adams wrote: > > Hi Hardik, > > The domain shape is not critical but the element shapes are. Your 100:1 domain aspect ratio is bad if you have N^3 mesh and thus element aspect ratios of 100:1. > If that is the case then you probably want to look at semi-coarsening multigrid. > > Mark > > On Fri, May 9, 2025 at 9:55?AM Hardik Kothari > wrote: >> Dear PETSc team, >> >> We are solving the Stokes equations using PETSc (via Firedrake) on a highly anisotropic 3D domain (L_x=1, L_y=0.01, L_z=0.1). >> >> In this setup, standard Schur complement preconditioners using a mass inverse for pressure struggle to converge. We could solve the problem with the LSC preconditioner (solver parameters are shown in the script). >> >> We have the following questions: >> Why standard preconditioners struggle in such domains? >> Why is the preconditioned residual norm for the Schur complement system much higher than the true residual norm? >> Would you recommend alternative or more robust preconditioners for such geometries? >> >> Thank you for your help. >> >> Best regards, >> Hardik >> >> >> >> >> HARDIK KOTHARI >> >> hardik.kothari at corintis.com >> >> Corintis SA >> EPFL Innovation Park Building C >> 1015 Lausanne >> >> >> >> >> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun May 11 15:00:20 2025 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 11 May 2025 16:00:20 -0400 Subject: [petsc-users] PETSc using problem In-Reply-To: References: <1e11e513.160ff6.196bfffb282.Coremail.hj2000@mail.ustc.edu.cn> Message-ID: I suppose we could call the postcheck even on failure. Then your first check in your postcheck would be on SNESLineSearchReason. What line search are you using? You could add the postcheck to all the early PetscFunctionReturns() in the line search routine. > On May 11, 2025, at 11:50?AM, Matthew Knepley wrote: > > On Sun, May 11, 2025 at 11:39?AM hj2000--- via petsc-users > wrote: >> Dear Developer: >> >> I am a user of PETSc, and I am using SNESLineSearchSetPostCheck to perform some actions when the line search fails. >> >> However, it seems that this function is only called when the line search is successful. Why is that? >> > PostCheck is intended to work on the solution found by the line search. > >> Do I need to do something else? >> > Yes. We do not have a specific hook for failure, because that would make it a different line search (one which may not obey the advertised properties). Thus we would want your scheme to be its own line search. > > Thanks, > > Matt > >> My PETSc version is 3.18.6. Below is my code. >> >> >> auto LineSearchPostCheck = []( SNESLineSearch ls, Vec x, Vec y, Vec w, PetscBool* changed_Y, >> >> PetscBool* changed_W, void* ) { >> >> SNESLineSearchReason lscnv; >> >> SNESLineSearchGetReason( ls, &lscnv ); >> >> SNES snes = nullptr; >> >> SNESLineSearchGetSNES( ls, &snes ); >> >> SNESConvergedReason cnv; >> >> SNESGetConvergedReason( snes, &cnv ); >> >> double lambda = 0.0; >> >> SNESLineSearchGetLambda( ls, &lambda ); >> >> PetscPrintf( MPI_COMM_WORLD, "lambda=%lf\n", lambda ); >> >> if ( lscnv != SNES_LINESEARCH_SUCCEEDED || lambda < 1e-4 ) >> >> { >> >> VecWAXPY( w, -0.1, y, x ); >> >> SNESLineSearchSetReason( ls, SNES_LINESEARCH_SUCCEEDED ); >> >> *changed_W = PETSC_TRUE; >> >> } >> >> return 0; >> >> }; >> >> SNESLineSearchSetPostCheck( ls, LineSearchPostCheck, nullptr); >> >> >> > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Z0il6G4Id0NXRUQd4JbC4hlU6LO5ceYj5jMmzoFHYBCPBWWyTvEkjNbe9jg3fc9Vo-zY-NaRmg9F_ywWSi_0kO8$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun May 11 16:11:53 2025 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 11 May 2025 17:11:53 -0400 Subject: [petsc-users] PETSc using problem In-Reply-To: References: <1e11e513.160ff6.196bfffb282.Coremail.hj2000@mail.ustc.edu.cn> Message-ID: On Sun, May 11, 2025 at 4:00?PM Barry Smith wrote: > > I suppose we could call the postcheck even on failure. Then your first > check in your postcheck would be on SNESLineSearchReason. What line search > are you using? You could add the postcheck to all the early > PetscFunctionReturns() in the line search routine. > As I said, I don't think I agree with PostCheck in this scenario. Matt > On May 11, 2025, at 11:50?AM, Matthew Knepley wrote: > > On Sun, May 11, 2025 at 11:39?AM hj2000--- via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Dear Developer: >> >> I am a user of PETSc, and I am using SNESLineSearchSetPostCheck to >> perform some actions when the line search fails. >> >> However, it seems that this function is only called when the line search >> is successful. Why is that? >> > PostCheck is intended to work on the solution found by the line search. > > >> Do I need to do something else? >> > Yes. We do not have a specific hook for failure, because that would make > it a different line search (one which may not obey the advertised > properties). Thus we would want your scheme to be its own line search. > > Thanks, > > Matt > > >> My PETSc version is 3.18.6. Below is my code. >> >> >> >> auto LineSearchPostCheck = []( SNESLineSearch ls, Vec x, Vec y, >> Vec w, PetscBool* changed_Y, >> >> PetscBool* changed_W, void* ) { >> >> SNESLineSearchReason lscnv; >> >> SNESLineSearchGetReason( ls, &lscnv ); >> >> SNES snes = nullptr; >> >> SNESLineSearchGetSNES( ls, &snes ); >> >> SNESConvergedReason cnv; >> >> SNESGetConvergedReason( snes, &cnv ); >> >> double lambda = 0.0; >> >> SNESLineSearchGetLambda( ls, &lambda ); >> >> PetscPrintf( MPI_COMM_WORLD, "lambda=%lf\n", lambda ); >> >> if ( lscnv != SNES_LINESEARCH_SUCCEEDED || lambda < 1e-4 ) >> >> { >> >> VecWAXPY( w, -0.1, y, x ); >> >> SNESLineSearchSetReason( ls, SNES_LINESEARCH_SUCCEEDED ); >> >> *changed_W = PETSC_TRUE; >> >> } >> >> return 0; >> >> }; >> >> SNESLineSearchSetPostCheck( ls, LineSearchPostCheck, nullptr); >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!deFm6anFdsGG0BfyGak4ewySR2Mh9Pquul08gXi74TZ_mScnmYbiUZXpoeJy_vLlOyZxW0yM2vZTz5m24eDN$ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!deFm6anFdsGG0BfyGak4ewySR2Mh9Pquul08gXi74TZ_mScnmYbiUZXpoeJy_vLlOyZxW0yM2vZTz5m24eDN$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dontbugthedevs at proton.me Mon May 12 16:29:35 2025 From: dontbugthedevs at proton.me (Noam T.) Date: Mon, 12 May 2025 21:29:35 +0000 Subject: [petsc-users] [C/Fortran] Calling DMPlexComputeCellGeometryFEM in a loop Message-ID: Hello, I ran into a seemingly non-consistent behavior of the function [DMPlexComputeCellGeometryFEM](https://urldefense.us/v3/__https://petsc.org/release/manualpages/DMPlex/DMPlexComputeCellGeometryFEM/__;!!G_uCfscf7eWS!YEv7_aaCdN0cxV8sUhRBIACYZGNNcwQusNjEpfVep6yOAxRYE9f8kw4XiVENrjEvQEVwiheebMTQVhPw4eenYApT_rJABo0_$ ) ( [DM](https://urldefense.us/v3/__https://petsc.org/release/manualpages/DM/DM/__;!!G_uCfscf7eWS!YEv7_aaCdN0cxV8sUhRBIACYZGNNcwQusNjEpfVep6yOAxRYE9f8kw4XiVENrjEvQEVwiheebMTQVhPw4eenYApT_kyYWLFi$ ) dm , [PetscInt](https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscInt/__;!!G_uCfscf7eWS!YEv7_aaCdN0cxV8sUhRBIACYZGNNcwQusNjEpfVep6yOAxRYE9f8kw4XiVENrjEvQEVwiheebMTQVhPw4eenYApT_jAVrwAd$ ) cell , [PetscQuadrature](https://urldefense.us/v3/__https://petsc.org/release/manualpages/DT/PetscQuadrature/__;!!G_uCfscf7eWS!YEv7_aaCdN0cxV8sUhRBIACYZGNNcwQusNjEpfVep6yOAxRYE9f8kw4XiVENrjEvQEVwiheebMTQVhPw4eenYApT_jVjnRvX$ ) quad , [PetscReal](https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscReal/__;!!G_uCfscf7eWS!YEv7_aaCdN0cxV8sUhRBIACYZGNNcwQusNjEpfVep6yOAxRYE9f8kw4XiVENrjEvQEVwiheebMTQVhPw4eenYApT_heQHCRx$ ) * v , [PetscReal](https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscReal/__;!!G_uCfscf7eWS!YEv7_aaCdN0cxV8sUhRBIACYZGNNcwQusNjEpfVep6yOAxRYE9f8kw4XiVENrjEvQEVwiheebMTQVhPw4eenYApT_heQHCRx$ ) J [], [PetscReal](https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscReal/__;!!G_uCfscf7eWS!YEv7_aaCdN0cxV8sUhRBIACYZGNNcwQusNjEpfVep6yOAxRYE9f8kw4XiVENrjEvQEVwiheebMTQVhPw4eenYApT_heQHCRx$ ) invJ [], [PetscReal](https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscReal/__;!!G_uCfscf7eWS!YEv7_aaCdN0cxV8sUhRBIACYZGNNcwQusNjEpfVep6yOAxRYE9f8kw4XiVENrjEvQEVwiheebMTQVhPw4eenYApT_heQHCRx$ ) * detJ ) Calling it in a loop happens to crash, only some times, when the input npoints is not equal to one. Attached is a mesh file (msh format), consisting of four quadrilaterals. Small examples in both C and Fortran also attached; they read the mesh, create a Gauss tensor quadrature rule and then loop through all cells in order to get the mapped quadrature points. When the quadrature created has a single point per dimension (npoints = 1), there seems to be no issue. The program runs and the output is the expected. On the C side, For npoints = 2 the program hangs, usually no errors; sometimes *** stack smashing detected ***: terminated, [:1345101] *** Process received signal *** [:1345101] Signal: Aborted (6) [:1345101] Signal code: (-6) or LeakSanitizer: detected memory leaks (when compiled with -fsanitize=addres). On the Fortran side, similar behavior for npoints = 1 or 2. Either no error at all (no error meaning the one that only mentions "probably memory access out of bounds"), or sometimes "corrupted size vs. prev_size", followed by a stack trace. Question: Do the arguments J, invJ need to be allocated (e.g. allocate(J(4))) before calling DMPlexComputeCellGeometryFEM? Allocation seems to be the cause for triggering an error. --- NOTE: The interface to DMPlexComputeCellGeometryFEM, under $PETSC_ARCH$/ftn/dm/petscdmplex.h90, read: interface DMPlexComputeCellGeometryFEM subroutine DMPlexComputeCellGeometryFEM(a,b,c,d,e,f,g, z) import tDM,tPetscQuadrature DM :: a PetscInt :: b PetscQuadrature :: c PetscReal :: d #### should it be d(*) ? PetscReal :: e(*) PetscReal :: f(*) PetscReal :: g PetscErrorCode z end subroutine end interface Should argument d (referring to the array of the mapped integration points v) be defined differently? --- NOTE: Tested with PETSc 3.23.1 (from the tarball) and the latest git. Configured with ./configure --with-fortran-bindings=1 Using gcc/gfortran 11.4.0, OpenMPI 4.1.2 Thanks, Noam -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mesh_quad.msh Type: model/mesh Size: 1441 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testF.F90 Type: text/x-fortran Size: 2290 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testC.c Type: text/x-csrc Size: 2455 bytes Desc: not available URL: From knepley at gmail.com Mon May 12 16:40:11 2025 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 12 May 2025 17:40:11 -0400 Subject: [petsc-users] [C/Fortran] Calling DMPlexComputeCellGeometryFEM in a loop In-Reply-To: References: Message-ID: On Mon, May 12, 2025 at 5:29?PM Noam T. via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello, > > I ran into a seemingly non-consistent behavior of the function > > DMPlexComputeCellGeometryFEM (DM dm, PetscInt cell, PetscQuadrature quad, PetscReal *v, PetscReal J[], PetscReal invJ[], PetscReal *detJ) > > Calling it in a loop happens to crash, only some times, when the input > npoints is not equal to one. > Attached is a mesh file (msh format), consisting of four quadrilaterals. > Small examples in both C and Fortran also attached; they read the mesh, > create a Gauss tensor quadrature rule and then loop through all cells in > order to get the mapped quadrature points. > Yes, this is what happens when interfaces evolve. When I originally wrote the function, I had only implemented affine geometry. When we implemented other coordinate spaces, the meaning of the arrays (x, J, JInv, detJ) changed. Now they held the evaluations at each quadrature point since they were not constant anymore. You need to allocate room for npoints copies of all of them. I need to fix the documentation. Thanks, Matt > When the quadrature created has a single point per dimension (npoints = 1), > there seems to be no issue. The program runs and the output is the > expected. > > On the C side, For npoints = 2 the program hangs, usually no errors; > sometimes > > *** stack smashing detected ***: terminated, > [:1345101] *** Process received signal *** > [:1345101] Signal: Aborted (6) > [:1345101] Signal code: (-6) > > or > > LeakSanitizer: detected memory leaks (when compiled with > -fsanitize=addres). > > On the Fortran side, similar behavior for npoints = 1 or 2. Either no > error at all (no error meaning the one that only mentions "probably > memory access out of bounds"), or sometimes "corrupted size vs. prev_size", > followed by a stack trace. > > Question: Do the arguments J, invJ need to be allocated (e.g. > allocate(J(4))) before calling DMPlexComputeCellGeometryFEM? Allocation > seems to be the cause for triggering an error. > > --- > > NOTE: The interface to DMPlexComputeCellGeometryFEM, under > $PETSC_ARCH$/ftn/dm/petscdmplex.h90, read: > > interface DMPlexComputeCellGeometryFEM > subroutine DMPlexComputeCellGeometryFEM(a,b,c,d,e,f,g, z) > import tDM,tPetscQuadrature > DM :: a > PetscInt :: b > PetscQuadrature :: c > PetscReal :: d #### should it be d(*) ? > PetscReal :: e(*) > PetscReal :: f(*) > PetscReal :: g > PetscErrorCode z > end subroutine > end interface > > Should argument d (referring to the array of the mapped integration > points v) be defined differently? > > --- > > NOTE: Tested with PETSc 3.23.1 (from the tarball) and the latest git. > Configured with ./configure --with-fortran-bindings=1 > Using gcc/gfortran 11.4.0, OpenMPI 4.1.2 > > Thanks, > Noam > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eAWWlW5LS8g4psK5Lef5MqtppJYejYI7g1LadD-JXWhiZVrx_jhz5HQ2T-VWxPh-ZdIPhN9KiG6PbssvMu8E$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ctchengben at mail.scut.edu.cn Tue May 13 09:39:31 2025 From: ctchengben at mail.scut.edu.cn (=?UTF-8?B?56iL5aWU?=) Date: Tue, 13 May 2025 22:39:31 +0800 (GMT+08:00) Subject: [petsc-users] Out of memory for using DMPlexCreateFromFile when load large-scale mesh Message-ID: <3767816a.2b840.196ca162683.Coremail.ctchengben@mail.scut.edu.cn> Hello, Recently I create unstructure mesh from Gmsh and its mesh format is msh file. However the mesh file contain around 100 million nodes, so when I use DMPlexCreateFromFile it only perform on a single CPU process thus out of memory. I ask GPT and it said PETSc may parallel perform DMPlexCreateFromFile when the mesh file is HDF5/Exodus II/XDMF, so it will decrease the Memory pressure for each CPU process. So I sent this email for asking help that how can I load such large-scale mesh to create DMPLEX. Looking forward to your reply! sinserely, Cheng. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hardik.kothari at corintis.com Tue May 13 10:13:18 2025 From: hardik.kothari at corintis.com (Hardik Kothari) Date: Tue, 13 May 2025 15:13:18 +0000 Subject: [petsc-users] Solving Stokes problem in high aspect ratio domains In-Reply-To: <5BF44380-C0B1-464B-BD56-F5000534EE52@joliv.et> References: <80f2df75-0faf-40ef-9409-a0422cfa7303.d0d10b7c-e93e-4ca5-8394-d4339a327c55.db30f155-f69a-4d1b-8a4d-6dafd7602ac1@emailsignatures365.codetwo.com> <4B3A1EE5-1A34-42B2-B36D-E89436F7EB25@corintis.com> <5BF44380-C0B1-464B-BD56-F5000534EE52@joliv.et> Message-ID: Dear Pierre, Dear PETSc team, Thank you for your response. In terms of geometry, we are moving toward more complex domains with more refined meshes that include multiple thin channels. We have been experimenting with the Stokes problem as a simplified case, but our main goal is to solve the high-Reynolds-number Navier?Stokes equations in these settings. We are currently planning to utilize a multi-node CPU architecture. For the Navier?Stokes system, we have experimented with both pressure-convection diffusion (PCD) and LSC preconditioners. In the thin channels, the PDC struggles to converge, and the LSC preconditioner is computationally slow, but it does converge eventually. Also, for both of these preconditioners for the thin channels, the norm of the preconditioned residual is much higher than the true residual norm, which likely indicates that neither preconditioner provides a sufficiently accurate approximation of the Schur complement. I would appreciate any insights you may have for better preconditioning strategies. Best regards, Hardik HARDIK KOTHARI hardik.kothari at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne [https://urldefense.us/v3/__https://storcor.s3.eu-central-1.amazonaws.com/logos/Logo-black.png__;!!G_uCfscf7eWS!ZkgL3p_ykjNhChJT714e-SsEGf_se8P03ZonnBbsSYm4XLTWR5khwNuHfc88CuP0c57UyPwjOWYcyMKkTLcXKGkCewbGGmUO$ ] Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure From: Pierre Jolivet Date: Sunday, 11 May 2025 at 20:45 To: Hardik Kothari Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Solving Stokes problem in high aspect ratio domains You don't often get email from pierre at joliv.et. Learn why this is important Do you want to refine the geometry or are you fine with the current one? What kind of hardware are you planning on using (GPU, single-node?)? Do you have a configuration for which LSC fails or does not give you good-enough performance? Thanks, Pierre On 9 May 2025, at 9:08?PM, Mark Adams wrote: Hi Hardik, The domain shape is not critical but the element shapes are. Your 100:1 domain aspect ratio is bad if you have N^3 mesh and thus element aspect ratios of 100:1. If that is the case then you probably want to look at semi-coarsening multigrid. Mark On Fri, May 9, 2025 at 9:55?AM Hardik Kothari > wrote: Dear PETSc team, We are solving the Stokes equations using PETSc (via Firedrake) on a highly anisotropic 3D domain (L_x=1, L_y=0.01, L_z=0.1). In this setup, standard Schur complement preconditioners using a mass inverse for pressure struggle to converge. We could solve the problem with the LSC preconditioner (solver parameters are shown in the script). We have the following questions: * Why standard preconditioners struggle in such domains? * Why is the preconditioned residual norm for the Schur complement system much higher than the true residual norm? * Would you recommend alternative or more robust preconditioners for such geometries? Thank you for your help. Best regards, Hardik HARDIK KOTHARI hardik.kothari at corintis.com Corintis SA EPFL Innovation Park Building C 1015 Lausanne [cid:~WRD0000.jpg] Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue May 13 10:40:16 2025 From: jed at jedbrown.org (Jed Brown) Date: Tue, 13 May 2025 09:40:16 -0600 Subject: [petsc-users] Solving Stokes problem in high aspect ratio domains In-Reply-To: References: <80f2df75-0faf-40ef-9409-a0422cfa7303.d0d10b7c-e93e-4ca5-8394-d4339a327c55.db30f155-f69a-4d1b-8a4d-6dafd7602ac1@emailsignatures365.codetwo.com> <4B3A1EE5-1A34-42B2-B36D-E89436F7EB25@corintis.com> <5BF44380-C0B1-464B-BD56-F5000534EE52@joliv.et> Message-ID: <87sel8tt7j.fsf@jedbrown.org> The preconditioned norm has units of state (like velocity and pressure) while the residual norm has units of residual. Note that both of these are dimensionally-inconsistent (they depend on the units/nondimensionalization you have chosen), and their relative scale also depends on units/nondimensionalization. So merely being "larger" doesn't mean the preconditioner isn't working. You'd need to assess whether the spectrum is better (of which, number of iterations to converge is a convenient surrogate and the thing you care about anyway). I would caution not to infer too much from Stokes if your interest is turbulence. The time step size is important for turbulence and diffusion is only of the same scale as advection. For Stokes, one doesn't normally use PCD because the inverse-viscosity weighted mass matrix is a spectrally equivalent preconditioner (and simpler than PCD). Hardik Kothari writes: > Dear Pierre, Dear PETSc team, > > Thank you for your response. > > In terms of geometry, we are moving toward more complex domains with more refined meshes that include multiple thin channels. We have been experimenting with the Stokes problem as a simplified case, but our main goal is to solve the high-Reynolds-number Navier?Stokes equations in these settings. > > We are currently planning to utilize a multi-node CPU architecture. > > For the Navier?Stokes system, we have experimented with both pressure-convection diffusion (PCD) and LSC preconditioners. In the thin channels, the PDC struggles to converge, and the LSC preconditioner is computationally slow, but it does converge eventually. Also, for both of these preconditioners for the thin channels, the norm of the preconditioned residual is much higher than the true residual norm, which likely indicates that neither preconditioner provides a sufficiently accurate approximation of the Schur complement. > > I would appreciate any insights you may have for better preconditioning strategies. > > Best regards, > Hardik > > > > HARDIK KOTHARI > > hardik.kothari at corintis.com > > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > > > [https://urldefense.us/v3/__https://storcor.s3.eu-central-1.amazonaws.com/logos/Logo-black.png__;!!G_uCfscf7eWS!ZkgL3p_ykjNhChJT714e-SsEGf_se8P03ZonnBbsSYm4XLTWR5khwNuHfc88CuP0c57UyPwjOWYcyMKkTLcXKGkCewbGGmUO$ ] > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure > From: Pierre Jolivet > Date: Sunday, 11 May 2025 at 20:45 > To: Hardik Kothari > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Solving Stokes problem in high aspect ratio domains > > You don't often get email from pierre at joliv.et. Learn why this is important > > Do you want to refine the geometry or are you fine with the current one? > What kind of hardware are you planning on using (GPU, single-node?)? > Do you have a configuration for which LSC fails or does not give you good-enough performance? > > Thanks, > Pierre > > > On 9 May 2025, at 9:08?PM, Mark Adams wrote: > > Hi Hardik, > > The domain shape is not critical but the element shapes are. Your 100:1 domain aspect ratio is bad if you have N^3 mesh and thus element aspect ratios of 100:1. > If that is the case then you probably want to look at semi-coarsening multigrid. > > Mark > > On Fri, May 9, 2025 at 9:55?AM Hardik Kothari > wrote: > Dear PETSc team, > > We are solving the Stokes equations using PETSc (via Firedrake) on a highly anisotropic 3D domain (L_x=1, L_y=0.01, L_z=0.1). > > In this setup, standard Schur complement preconditioners using a mass inverse for pressure struggle to converge. We could solve the problem with the LSC preconditioner (solver parameters are shown in the script). > > We have the following questions: > > * Why standard preconditioners struggle in such domains? > * Why is the preconditioned residual norm for the Schur complement system much higher than the true residual norm? > * Would you recommend alternative or more robust preconditioners for such geometries? > > Thank you for your help. > > Best regards, > Hardik > > > > HARDIK KOTHARI > > hardik.kothari at corintis.com > > Corintis SA > EPFL Innovation Park Building C > 1015 Lausanne > > > > > [cid:~WRD0000.jpg] > > Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure From pierre at joliv.et Tue May 13 23:23:49 2025 From: pierre at joliv.et (Pierre Jolivet) Date: Wed, 14 May 2025 06:23:49 +0200 Subject: [petsc-users] Solving Stokes problem in high aspect ratio domains In-Reply-To: <87sel8tt7j.fsf@jedbrown.org> References: <80f2df75-0faf-40ef-9409-a0422cfa7303.d0d10b7c-e93e-4ca5-8394-d4339a327c55.db30f155-f69a-4d1b-8a4d-6dafd7602ac1@emailsignatures365.codetwo.com> <4B3A1EE5-1A34-42B2-B36D-E89436F7EB25@corintis.com> <5BF44380-C0B1-464B-BD56-F5000534EE52@joliv.et> <87sel8tt7j.fsf@jedbrown.org> Message-ID: <88A5F153-F2EE-4C58-B7DE-CD549D10DF2D@joliv.et> > On 13 May 2025, at 5:40?PM, Jed Brown wrote: > > The preconditioned norm has units of state (like velocity and pressure) while the residual norm has units of residual. Note that both of these are dimensionally-inconsistent (they depend on the units/nondimensionalization you have chosen), and their relative scale also depends on units/nondimensionalization. So merely being "larger" doesn't mean the preconditioner isn't working. You'd need to assess whether the spectrum is better (of which, number of iterations to converge is a convenient surrogate and the thing you care about anyway). > > I would caution not to infer too much from Stokes if your interest is turbulence. The time step size is important for turbulence and diffusion is only of the same scale as advection. For Stokes, one doesn't normally use PCD because the inverse-viscosity weighted mass matrix is a spectrally equivalent preconditioner (and simpler than PCD). That being said, if you have something closer to your target discretization/geometry that you?d be willing to share, some of us (or at the very least, me) would be happy to have a look. I understand you may not want to share either of these publicly if this is part of a closed-source topology optimization framework, in which case you can send stuff at petsc-maint at mcs.anl.gov Thanks, Pierre PS: I?ve tried some of my stuff on your problem (either monolithically or as part of PCFIELDSPLIT), they trivially solve it (even with a slightly refined geometry, notice the number of unknowns), but they have a high setup cost, and I can?t guarantee they?ll work for your final problem, as Jed mentioned. DOFs: 2137103 Residual norms for firedrake_0_ solve. 0 KSP unpreconditioned resid norm 4.898001545440e-03 true resid norm 4.898001545440e-03 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 4.897530535125e-03 true resid norm 4.897530535125e-03 ||r(i)||/||b|| 9.999038362258e-01 2 KSP unpreconditioned resid norm 4.895485358665e-03 true resid norm 4.895485358665e-03 ||r(i)||/||b|| 9.994862829765e-01 3 KSP unpreconditioned resid norm 4.892258200545e-03 true resid norm 4.892258200545e-03 ||r(i)||/||b|| 9.988274105589e-01 4 KSP unpreconditioned resid norm 4.832684872769e-03 true resid norm 4.832684872769e-03 ||r(i)||/||b|| 9.866646279988e-01 5 KSP unpreconditioned resid norm 3.484018420031e-03 true resid norm 3.484018420031e-03 ||r(i)||/||b|| 7.113142753650e-01 6 KSP unpreconditioned resid norm 1.317379071184e-03 true resid norm 1.317379071184e-03 ||r(i)||/||b|| 2.689625674803e-01 7 KSP unpreconditioned resid norm 2.393153081654e-04 true resid norm 2.393153081654e-04 ||r(i)||/||b|| 4.885978616895e-02 8 KSP unpreconditioned resid norm 4.276785206897e-05 true resid norm 4.276785206898e-05 ||r(i)||/||b|| 8.731694278210e-03 9 KSP unpreconditioned resid norm 7.000151646440e-06 true resid norm 7.000151646437e-06 ||r(i)||/||b|| 1.429185266990e-03 10 KSP unpreconditioned resid norm 1.407327852370e-06 true resid norm 1.407327852358e-06 ||r(i)||/||b|| 2.873269514723e-04 11 KSP unpreconditioned resid norm 2.223672507944e-07 true resid norm 2.223672507690e-07 ||r(i)||/||b|| 4.539958771063e-05 12 KSP unpreconditioned resid norm 4.643025030927e-08 true resid norm 4.643025033995e-08 ||r(i)||/||b|| 9.479427458160e-06 13 KSP unpreconditioned resid norm 9.835866346159e-09 true resid norm 9.835866346336e-09 ||r(i)||/||b|| 2.008138677599e-06 14 KSP unpreconditioned resid norm 1.658715911665e-09 true resid norm 1.658715908692e-09 ||r(i)||/||b|| 3.386515690745e-07 15 KSP unpreconditioned resid norm 3.674829203051e-10 true resid norm 3.674829226619e-10 ||r(i)||/||b|| 7.502711447776e-08 16 KSP unpreconditioned resid norm 6.299503951584e-11 true resid norm 6.299505159177e-11 ||r(i)||/||b|| 1.286137846372e-08 17 KSP unpreconditioned resid norm 1.078827517736e-11 true resid norm 1.078831394356e-11 ||r(i)||/||b|| 2.202595046872e-09 KSP final norm of residual 1.07883e-11 > Hardik Kothari > writes: > >> Dear Pierre, Dear PETSc team, >> >> Thank you for your response. >> >> In terms of geometry, we are moving toward more complex domains with more refined meshes that include multiple thin channels. We have been experimenting with the Stokes problem as a simplified case, but our main goal is to solve the high-Reynolds-number Navier?Stokes equations in these settings. >> >> We are currently planning to utilize a multi-node CPU architecture. >> >> For the Navier?Stokes system, we have experimented with both pressure-convection diffusion (PCD) and LSC preconditioners. In the thin channels, the PDC struggles to converge, and the LSC preconditioner is computationally slow, but it does converge eventually. Also, for both of these preconditioners for the thin channels, the norm of the preconditioned residual is much higher than the true residual norm, which likely indicates that neither preconditioner provides a sufficiently accurate approximation of the Schur complement. >> >> I would appreciate any insights you may have for better preconditioning strategies. >> >> Best regards, >> Hardik >> >> >> >> HARDIK KOTHARI >> >> hardik.kothari at corintis.com >> >> Corintis SA >> EPFL Innovation Park Building C >> 1015 Lausanne >> >> >> >> [https://urldefense.us/v3/__https://storcor.s3.eu-central-1.amazonaws.com/logos/Logo-black.png__;!!G_uCfscf7eWS!ZkgL3p_ykjNhChJT714e-SsEGf_se8P03ZonnBbsSYm4XLTWR5khwNuHfc88CuP0c57UyPwjOWYcyMKkTLcXKGkCewbGGmUO$ ] >> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure >> From: Pierre Jolivet > >> Date: Sunday, 11 May 2025 at 20:45 >> To: Hardik Kothari > >> Cc: petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] Solving Stokes problem in high aspect ratio domains >> >> You don't often get email from pierre at joliv.et . Learn why this is important >> >> Do you want to refine the geometry or are you fine with the current one? >> What kind of hardware are you planning on using (GPU, single-node?)? >> Do you have a configuration for which LSC fails or does not give you good-enough performance? >> >> Thanks, >> Pierre >> >> >> On 9 May 2025, at 9:08?PM, Mark Adams > wrote: >> >> Hi Hardik, >> >> The domain shape is not critical but the element shapes are. Your 100:1 domain aspect ratio is bad if you have N^3 mesh and thus element aspect ratios of 100:1. >> If that is the case then you probably want to look at semi-coarsening multigrid. >> >> Mark >> >> On Fri, May 9, 2025 at 9:55?AM Hardik Kothari > wrote: >> Dear PETSc team, >> >> We are solving the Stokes equations using PETSc (via Firedrake) on a highly anisotropic 3D domain (L_x=1, L_y=0.01, L_z=0.1). >> >> In this setup, standard Schur complement preconditioners using a mass inverse for pressure struggle to converge. We could solve the problem with the LSC preconditioner (solver parameters are shown in the script). >> >> We have the following questions: >> >> * Why standard preconditioners struggle in such domains? >> * Why is the preconditioned residual norm for the Schur complement system much higher than the true residual norm? >> * Would you recommend alternative or more robust preconditioners for such geometries? >> >> Thank you for your help. >> >> Best regards, >> Hardik >> >> >> >> HARDIK KOTHARI >> >> hardik.kothari at corintis.com >> >> Corintis SA >> EPFL Innovation Park Building C >> 1015 Lausanne >> >> >> >> >> [cid:~WRD0000.jpg] >> >> Here at Corintis we care for your privacy. That is why we have taken appropriate measures to ensure that the data you have provided to us is always secure -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed May 14 05:19:30 2025 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 14 May 2025 06:19:30 -0400 Subject: [petsc-users] Out of memory for using DMPlexCreateFromFile when load large-scale mesh In-Reply-To: <3767816a.2b840.196ca162683.Coremail.ctchengben@mail.scut.edu.cn> References: <3767816a.2b840.196ca162683.Coremail.ctchengben@mail.scut.edu.cn> Message-ID: There are examples listed at the end of the *DMPlexCreateFromFile ,*and the first two are designed for parallel file load-store. Matt may have a better idea but I would test that you can run these tests (there are test arguments in a comment at the end of the source file), to get your installation working, and then clone that code. Mark On Tue, May 13, 2025 at 10:40?AM ?? wrote: > Hello, > > Recently I create unstructure mesh from Gmsh and its mesh format is msh > file. However the mesh file contain around 100 million nodes, so when I > use *DMPlexCreateFromFile > * > it only perform on a single CPU process thus out of memory. > > I ask GPT and it said PETSc may parallel perform *DMPlexCreateFromFile > *when > the mesh file is HDF5/Exodus II/XDMF, so it will decrease the Memory > pressure for each CPU process. > > So I sent this email for asking help that how can I load such large-scale > mesh to create DMPLEX. > > > Looking forward to your reply! > > > sinserely, > Cheng. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed May 14 06:24:48 2025 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 14 May 2025 07:24:48 -0400 Subject: [petsc-users] Out of memory for using DMPlexCreateFromFile when load large-scale mesh In-Reply-To: References: <3767816a.2b840.196ca162683.Coremail.ctchengben@mail.scut.edu.cn> Message-ID: On Wed, May 14, 2025 at 6:19?AM Mark Adams wrote: > There are examples listed at the end of the *DMPlexCreateFromFile > ,*and > the first two are designed for parallel file load-store. > Matt may have a better idea but I would test that you can run these tests > (there are test arguments in a comment at the end of the source file), to > get your installation working, and then clone that code. > On the machine on which you created the GMsh file there is definitely enough memory. Thus I would load it there and save it as HDF5 or NetCDF, so that you could load it in parallel elsewhere. For HDF5, I would use the latest formet -dm_plex_view_hdf5_storage_version 3.1.0 Thanks, Matt > Mark > > On Tue, May 13, 2025 at 10:40?AM ?? wrote: > >> Hello, >> >> Recently I create unstructure mesh from Gmsh and its mesh format is msh >> file. However the mesh file contain around 100 million nodes, so when I >> use *DMPlexCreateFromFile >> * >> it only perform on a single CPU process thus out of memory. >> >> I ask GPT and it said PETSc may parallel perform *DMPlexCreateFromFile >> *when >> the mesh file is HDF5/Exodus II/XDMF, so it will decrease the Memory >> pressure for each CPU process. >> >> So I sent this email for asking help that how can I load such large-scale >> mesh to create DMPLEX. >> >> >> Looking forward to your reply! >> >> >> sinserely, >> Cheng. >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZZx-2RTs8T5ThExGXILH1XXU3-8Ko9JD4p9a85zHm9D1bn8xxrag2HfMb0zD5jAYQJWiL6CyI62KgWl0TgHb$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ctchengben at mail.scut.edu.cn Thu May 15 08:15:36 2025 From: ctchengben at mail.scut.edu.cn (=?UTF-8?B?56iL5aWU?=) Date: Thu, 15 May 2025 21:15:36 +0800 (GMT+08:00) Subject: [petsc-users] Error in configuring HDF5 with Cygwin on Windows by using Intel MPI Message-ID: <35e1e60e.2beff.196d4160a75.Coremail.ctchengben@mail.scut.edu.cn> Hello, Recently I try successfully to install PETSc with Cygwin and Visual Studio on Windows10 plateform(with external packages metis and parmetis). Now I want to use hdf5, so I re-configure the PETSc on the cygwin with Native Microsoft/Intel Windows Compilers. The softwares/packages used below: 1. PETSc: version 3.23.2 2. VS: version 2022 3. Intel MPI: download Intel oneAPI Base Toolkit and HPC Toolkit 4. Cygwin And the compiler option in configuration is: ./configure --with-debugging=0 --with-cc=cl --with-fc=ifort --with-cxx=cl --with-blaslapack-lib=-L/cygdrive/g/Intel/oneAPI/mkl/2023.2.0/lib/intel64 mkl-intel-lp64-dll.lib mkl-sequential-dll.lib mkl-core-dll.lib --download-fblaslapack=/cygdrive/g/mypetsc/petsc-pkg-fblaslapack-e8a03f57d64c.tar.gz --with-mpi-include=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/include --with-mpi-lib=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/lib/release/impi.lib --with-mpiexec=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/bin/mpiexec -localonly --download-metis=/cygdrive/g/mypetsc/petsc-pkg-metis-8b194fdf0966.tar.gz --download-parmetis=/cygdrive/g/mypetsc/petsc-pkg-parmetis-f5e3aab04fd5.tar.gz --with-strict-petscerrorcode=0 --with-64-bit-indices --download-hdf5=/cygdrive/g/mypetsc/hdf5-1.14.3-p1.tar.bz2 --download-zlib=/cygdrive/g/mypetsc/zlib-1.3.1.tar.gz However it return an error: UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): --------------------------------------------------------------------------------------------- External package zlib does not support --download-zlib with Microsoft compilers ********************************************************************************************* --------------------------------------------------------------------------------------------- External package hdf5 does not support --download-hdf5 with Microsoft compilers The configure.log is attached. It seem that I can't use Microsoft/Intel Windows Compilers to install hdf5, and Libraries built with Cygwin/GNU compilers are not compatible and cannot be linked with Microsoft or Intel compilers. But I do use Intel compiler on the Visual studio. So I wrrit this email to report my problem and ask for your help that how can I use PETSc with hdf5 on the Windows. Looking forward your reply! sinserely, Cheng. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: configure.log URL: From knepley at gmail.com Thu May 15 08:35:04 2025 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 15 May 2025 09:35:04 -0400 Subject: [petsc-users] Error in configuring HDF5 with Cygwin on Windows by using Intel MPI In-Reply-To: <35e1e60e.2beff.196d4160a75.Coremail.ctchengben@mail.scut.edu.cn> References: <35e1e60e.2beff.196d4160a75.Coremail.ctchengben@mail.scut.edu.cn> Message-ID: On Thu, May 15, 2025 at 9:16?AM ?? wrote: > Hello, > Recently I try successfully to install PETSc with Cygwin and Visual Studio > on Windows10 plateform(with external packages metis and parmetis). > > Now I want to use hdf5, so I re-configure the PETSc on the cygwin with Native > Microsoft/Intel Windows Compilers. > The softwares/packages used below: > 1. PETSc: version 3.23.2 > 2. VS: version 2022 > 3. Intel MPI: download Intel oneAPI Base Toolkit and HPC Toolkit > 4. Cygwin > > And the compiler option in configuration is: > ./configure --with-debugging=0 --with-cc=cl --with-fc=ifort > --with-cxx=cl > --with-blaslapack-lib=-L/cygdrive/g/Intel/oneAPI/mkl/2023.2.0/lib/intel64 > mkl-intel-lp64-dll.lib mkl-sequential-dll.lib mkl-core-dll.lib > --download-fblaslapack=/cygdrive/g/mypetsc/petsc-pkg-fblaslapack-e8a03f57d64c.tar.gz > --with-mpi-include=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/include > --with-mpi-lib=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/lib/release/impi.lib > --with-mpiexec=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/bin/mpiexec > -localonly > --download-metis=/cygdrive/g/mypetsc/petsc-pkg-metis-8b194fdf0966.tar.gz > --download-parmetis=/cygdrive/g/mypetsc/petsc-pkg-parmetis-f5e3aab04fd5.tar.gz > --with-strict-petscerrorcode=0 --with-64-bit-indices > --download-hdf5=/cygdrive/g/mypetsc/hdf5-1.14.3-p1.tar.bz2 > --download-zlib=/cygdrive/g/mypetsc/zlib-1.3.1.tar.gz > > > > However it return an error: > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > --------------------------------------------------------------------------------------------- > External package zlib does not support --download-zlib with > Microsoft compilers > ********************************************************************************************* > > > --------------------------------------------------------------------------------------------- > External package hdf5 does not support --download-hdf5 with > Microsoft compilers > > > The configure.log is attached. > > > It seem that I can't use Microsoft/Intel Windows Compilers to install > hdf5, and > > - > > Libraries built with Cygwin/GNU compilers are not compatible and > cannot be linked with Microsoft or Intel compilers. But I do use > Intel compiler on the Visual studio. > > > > > So I wrrit this email to report my problem and ask for your help that how > can I use PETSc with hdf5 on the Windows. > 1. You can build under WSL2, which is what I would recommend. 2. If you really want to use Windows compilers, you will have to build HDF5 yourself. Thanks, Matt > Looking forward your reply! > > > sinserely, > Cheng. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YEvzHiQxBu4jH7P_5mh66tM7zYwZp9qXzbyMwybgRUfsrNT9wBHxgW8-2A69zpap_nYgRiP51tb3JrWE7kke$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dontbugthedevs at proton.me Mon May 19 17:57:33 2025 From: dontbugthedevs at proton.me (Noam T.) Date: Mon, 19 May 2025 22:57:33 +0000 Subject: [petsc-users] Element connectivity of a DMPlex Message-ID: Hello, I am trying to build the connectivity matrix for a mesh; i.e. the indices of the nodes that compose each cell. Example: 0-------1 |.\.....| |...\...| |.....\.| 3-------2 the matrix would look like [ [0, 3, 2], [2, 1, 0] ] (possibly different ordering). One option is using DMPlexGetTransitiveClosure, and accessing the last elements in the output "points", which contain the vertex indices. However, these indices are local per process (both for vertices and cells). Is it possible to get the global indices? I tried a mapping, as in example 14f (https://urldefense.us/v3/__https://petsc.org/release/src/ksp/ksp/tutorials/ex14f.F90.html__;!!G_uCfscf7eWS!ezMKlDED-8WxXpg_Elxus7WhYkZzCA5OmmLe6vHZGFoj4se5S5RD5ZgTrC3TYErx8znYKebSW-s0H0mt1_MEMAzpmFDwr0PH$ ) --- DM :: dm ISLocalToGlobalMapping :: ltog_map PetscInt, pointer :: ltog_idx(:) ! ... ! Create a dm from a mesh file ! ... DMGetLocalToGlobalMapping(dm,ltog_map,ierr) ISLocalToGlobalMappingGetIndices(ltog_map, ltog_idx, ierr) --- but the returned array ltog is empty (ISLocalToGlobalMappingGetSize() returns zero). Are there other functions calls needed before being able to create this mapping? Or is this mapping simply not usable in this case? Is there perhaps better/simpler way to get this connectivity? Thank you. Noam -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed May 21 02:42:51 2025 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 21 May 2025 03:42:51 -0400 Subject: [petsc-users] Element connectivity of a DMPlex In-Reply-To: References: Message-ID: Hi Noam, I think you want https://urldefense.us/v3/__https://petsc.org/release/manualpages/DMPlex/DMPlexGetClosureIndices/__;!!G_uCfscf7eWS!f8v1_ty7e_mf4QhytGsCFXSd_APEy50GjoLrAL_o294K-EBz1Xa8kEDmeq3UmehbZvBL-u72r4GN6mjxAFf6Ljw$ Indices are ordered globally on the first process (0) first, the proc 1, etc., so the global index is the local index + "my start index". You can get this local start index in a number of ways, eg, https://urldefense.us/v3/__https://petsc.org/main/manualpages/Mat/MatGetOwnershipRange__;!!G_uCfscf7eWS!f8v1_ty7e_mf4QhytGsCFXSd_APEy50GjoLrAL_o294K-EBz1Xa8kEDmeq3UmehbZvBL-u72r4GN6mjxh3g3zew$ if you have a scalar matrix from the DM/section. Thanks, Mark ps, we are at the PETSc annual meeting this week, so may be slow to respond On Mon, May 19, 2025 at 6:58?PM Noam T. via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello, > > I am trying to build the connectivity matrix for a mesh; i.e. the indices > of the nodes that compose each cell. > > Example: > > 0-------1 > |.\.....| > |...\...| > |.....\.| > 3-------2 > > the matrix would look like [ [0, 3, 2], [2, 1, 0] ] (possibly different > ordering). > > One option is using DMPlexGetTransitiveClosure, and accessing the last > elements in the output "points", which contain the vertex indices. > However, these indices are local per process (both for vertices and > cells). Is it possible to get the global indices? > > I tried a mapping, as in example 14f ( > https://urldefense.us/v3/__https://petsc.org/release/src/ksp/ksp/tutorials/ex14f.F90.html__;!!G_uCfscf7eWS!f8v1_ty7e_mf4QhytGsCFXSd_APEy50GjoLrAL_o294K-EBz1Xa8kEDmeq3UmehbZvBL-u72r4GN6mjxUrZKlTM$ > > ) > > --- > DM :: dm > ISLocalToGlobalMapping :: ltog_map > PetscInt, pointer :: ltog_idx(:) > > ! ... > ! Create a dm from a mesh file > ! ... > DMGetLocalToGlobalMapping(dm,ltog_map,ierr) > ISLocalToGlobalMappingGetIndices(ltog_map, ltog_idx, ierr) > --- > > but the returned array ltog is empty (ISLocalToGlobalMappingGetSize() > returns zero). Are there other functions calls needed before being able > to create this mapping? Or is this mapping simply not usable in this case? > > Is there perhaps better/simpler way to get this connectivity? > > Thank you. > Noam > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From edoardo.centofanti01 at universitadipavia.it Wed May 21 03:40:48 2025 From: edoardo.centofanti01 at universitadipavia.it (Edoardo Centofanti) Date: Wed, 21 May 2025 10:40:48 +0200 Subject: [petsc-users] Questions about PCEXOTIC Message-ID: Dear all, I have some questions about the PCEXOTIC preconditioner: 1- While the type "wirebasket" seems clear to me, I do not understand what is meant as "one dof per face" in the documentation referring to "face" type. Is it intended as the interpolation on the centre of the face or some mean value? 2- Is it possible to change the overlap between the subdomains? 3- Are there any plans of extending the implementation to unstructured grids (DMPlex-like)? The current one seems to rely heavily on DMDA 4- Running some preliminary tests on GPU with a code of mine that works on CPU, I got the following error: ** On entry to cusparseSpMM_bufferSize(): dimension mismatch, matA.num_rows (15405) != matC.num_rows (44955) with opA = CUSPARSE_OPERATION_NON_TRANSPOSE Is the preconditioner intended to run also on GPU or the error above comes from the fact that this feature has yet to be tested/fully implemented? Thank you in advance, Best, Edoardo -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at joliv.et Wed May 21 03:51:53 2025 From: pierre at joliv.et (Pierre Jolivet) Date: Wed, 21 May 2025 10:51:53 +0200 Subject: [petsc-users] Questions about PCEXOTIC In-Reply-To: References: Message-ID: <3C3F282B-8E28-4831-8955-14E122808C87@joliv.et> > On 21 May 2025, at 10:41?AM, Edoardo Centofanti wrote: > > ? > Dear all, > > I have some questions about the PCEXOTIC preconditioner: > > 1- While the type "wirebasket" seems clear to me, I do not understand what is meant as "one dof per face" in the documentation referring to "face" type. Is it intended as the interpolation on the centre of the face or some mean value? > > 2- Is it possible to change the overlap between the subdomains? > > 3- Are there any plans of extending the implementation to unstructured grids (DMPlex-like)? The current one seems to rely heavily on DMDA > > 4- Running some preliminary tests on GPU with a code of mine that works on CPU, I got the following error: > > ** On entry to cusparseSpMM_bufferSize(): dimension mismatch, matA.num_rows (15405) != matC.num_rows (44955) with opA = CUSPARSE_OPERATION_NON_TRANSPOSE > > Is the preconditioner intended to run also on GPU or the error above comes from the fact that this feature has yet to be tested/fully implemented? > This preconditioner is very old and the code is barely maintained. There is a newer implementation of the GDSW preconditioner in PCMG, see https://urldefense.us/v3/__https://petsc.org/release/manualpages/PC/PCMGSetAdaptCoarseSpaceType/__;!!G_uCfscf7eWS!ZDnv3udGFNlZEC_MRfLAqwvcvCObnaAjMCeyaW__UaecBvN3MkXrzOPwJJUSHXo80-xOQdLNnSf_nLmUd7bZxg$ Thanks, Pierre > Thank you in advance, > Best, > Edoardo > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed May 21 07:50:05 2025 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 21 May 2025 08:50:05 -0400 Subject: [petsc-users] Element connectivity of a DMPlex In-Reply-To: References: Message-ID: On Mon, May 19, 2025 at 6:57?PM Noam T. via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello, > > I am trying to build the connectivity matrix for a mesh; i.e. the indices > of the nodes that compose each cell. > > Example: > > 0-------1 > |.\.....| > |...\...| > |.....\.| > 3-------2 > > the matrix would look like [ [0, 3, 2], [2, 1, 0] ] (possibly different > ordering). > > One option is using DMPlexGetTransitiveClosure, and accessing the last > elements in the output "points", which contain the vertex indices. > However, these indices are local per process (both for vertices and > cells). Is it possible to get the global indices? > > I tried a mapping, as in example 14f ( > https://urldefense.us/v3/__https://petsc.org/release/src/ksp/ksp/tutorials/ex14f.F90.html__;!!G_uCfscf7eWS!cZBA-cU24g0I7WDWgJbyMqwJKxk50DrkIGAu36vgS0MNYlEAzNA354o5U-YVkcur8NsRsoSXABuuOsuln5el$ > > ) > > --- > DM :: dm > ISLocalToGlobalMapping :: ltog_map > PetscInt, pointer :: ltog_idx(:) > > ! ... > ! Create a dm from a mesh file > ! ... > DMGetLocalToGlobalMapping(dm,ltog_map,ierr) > ISLocalToGlobalMappingGetIndices(ltog_map, ltog_idx, ierr) > --- > > but the returned array ltog is empty (ISLocalToGlobalMappingGetSize() > returns zero). Are there other functions calls needed before being able > to create this mapping? Or is this mapping simply not usable in this case? > > Is there perhaps better/simpler way to get this connectivity? > There are several ways you might do this. It helps to know what you are aiming for. 1) If you just want this output, it might be easier to just DMView() with the PETSC_VIEWER_HDF5_VIZ format, since that just outputs the cell-vertex topology and coordinates 2) You can call DMPlexUninterpolate() to produce a mesh with just cells and vertices, and output it in any format. 3) If you want it in memory, but still with global indices (I don't understand this use case), then you can use DMPlexCreatePointNumbering() for an overall global numbering, or DMPlexCreateCellNumbering() and DMPlexCreateVertexNumbering() for separate global numberings. Thanks, Matt > Thank you. > Noam > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cZBA-cU24g0I7WDWgJbyMqwJKxk50DrkIGAu36vgS0MNYlEAzNA354o5U-YVkcur8NsRsoSXABuuOvSIT50e$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed May 21 09:52:28 2025 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 21 May 2025 10:52:28 -0400 Subject: [petsc-users] Questions about PCEXOTIC In-Reply-To: <3C3F282B-8E28-4831-8955-14E122808C87@joliv.et> References: <3C3F282B-8E28-4831-8955-14E122808C87@joliv.et> Message-ID: <2CF3E79F-09C2-4A77-AEC2-82B969DC6181@petsc.dev> Thanks for your interest in this type of algorithm. As Pierre notes, the code is implemented only as a simple tutorial on one particular "exotic" wire basket-type algorithm. We don't recommend this implementation for more general algorithms or geometries. The face dof represents some sort of "average" of the face solution, it is not a point value. Barry > On May 21, 2025, at 4:51?AM, Pierre Jolivet wrote: > > > >> On 21 May 2025, at 10:41?AM, Edoardo Centofanti wrote: >> >> ? >> Dear all, >> >> I have some questions about the PCEXOTIC preconditioner: >> >> 1- While the type "wirebasket" seems clear to me, I do not understand what is meant as "one dof per face" in the documentation referring to "face" type. Is it intended as the interpolation on the centre of the face or some mean value? >> >> 2- Is it possible to change the overlap between the subdomains? >> >> 3- Are there any plans of extending the implementation to unstructured grids (DMPlex-like)? The current one seems to rely heavily on DMDA >> >> 4- Running some preliminary tests on GPU with a code of mine that works on CPU, I got the following error: >> >> ** On entry to cusparseSpMM_bufferSize(): dimension mismatch, matA.num_rows (15405) != matC.num_rows (44955) with opA = CUSPARSE_OPERATION_NON_TRANSPOSE >> >> Is the preconditioner intended to run also on GPU or the error above comes from the fact that this feature has yet to be tested/fully implemented? >> > > This preconditioner is very old and the code is barely maintained. > There is a newer implementation of the GDSW preconditioner in PCMG, see https://urldefense.us/v3/__https://petsc.org/release/manualpages/PC/PCMGSetAdaptCoarseSpaceType/__;!!G_uCfscf7eWS!ZumKWSWq6NO-AQvGkbYva1mSwRd9bF0GiINdu1CHcu0VVODx8UduTv-c85YTwWQ9OAUxqtHsbIcrIBgM5xKHUvE$ > > Thanks, > Pierre > >> Thank you in advance, >> Best, >> Edoardo >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From edoardo.centofanti01 at universitadipavia.it Thu May 22 02:58:32 2025 From: edoardo.centofanti01 at universitadipavia.it (Edoardo Centofanti) Date: Thu, 22 May 2025 09:58:32 +0200 Subject: [petsc-users] Questions about PCEXOTIC In-Reply-To: <2CF3E79F-09C2-4A77-AEC2-82B969DC6181@petsc.dev> References: <3C3F282B-8E28-4831-8955-14E122808C87@joliv.et> <2CF3E79F-09C2-4A77-AEC2-82B969DC6181@petsc.dev> Message-ID: Dear all, Thank you for your answers. Do you know which options are required for a typical use case of GDSW in PCMG? However, I imagine that only structured DMDA meshes can be employed with it. Best, Edoardo Il giorno mer 21 mag 2025 alle ore 16:52 Barry Smith ha scritto: > > Thanks for your interest in this type of algorithm. > > As Pierre notes, the code is implemented only as a simple tutorial on > one particular "exotic" wire basket-type algorithm. We don't recommend this > implementation for more general algorithms or geometries. > > The face dof represents some sort of "average" of the face solution, it > is not a point value. > > Barry > > > On May 21, 2025, at 4:51?AM, Pierre Jolivet wrote: > > > > On 21 May 2025, at 10:41?AM, Edoardo Centofanti < > edoardo.centofanti01 at universitadipavia.it> wrote: > > ? > Dear all, > > I have some questions about the PCEXOTIC preconditioner: > > 1- While the type "wirebasket" seems clear to me, I do not understand what > is meant as "one dof per face" in the documentation referring to "face" > type. Is it intended as the interpolation on the centre of the face or some > mean value? > > 2- Is it possible to change the overlap between the subdomains? > > 3- Are there any plans of extending the implementation to unstructured > grids (DMPlex-like)? The current one seems to rely heavily on DMDA > > 4- Running some preliminary tests on GPU with a code of mine that works on > CPU, I got the following error: > > ** On entry to cusparseSpMM_bufferSize(): dimension mismatch, > matA.num_rows (15405) != matC.num_rows (44955) with opA = > CUSPARSE_OPERATION_NON_TRANSPOSE > > Is the preconditioner intended to run also on GPU or the error above comes > from the fact that this feature has yet to be tested/fully implemented? > > > This preconditioner is very old and the code is barely maintained. > There is a newer implementation of the GDSW preconditioner in PCMG, see > https://urldefense.us/v3/__https://petsc.org/release/manualpages/PC/PCMGSetAdaptCoarseSpaceType/__;!!G_uCfscf7eWS!Zmi6sdgeoCOMy1lkVGYn-ac5wvH3QEcOkYQ2VzdQgCqLY8nIVkX1oi_yt5S2vJDcF5BT8mmzOLmOUEK483D0v9_aZe01iy1b20hMB1HPuOE$ > > > Thanks, > Pierre > > Thank you in advance, > Best, > Edoardo > > > -- Edoardo Centofanti Dipartimento di Matematica 'Felice Casorati' Universit? degli Studi di Pavia Tel. 0382985608 | Ufficio A14 | Via Ferrata 5, 27100 Pavia, Italy -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Thu May 22 03:33:25 2025 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Thu, 22 May 2025 11:33:25 +0300 Subject: [petsc-users] Questions about PCEXOTIC In-Reply-To: References: <3C3F282B-8E28-4831-8955-14E122808C87@joliv.et> <2CF3E79F-09C2-4A77-AEC2-82B969DC6181@petsc.dev> Message-ID: GDSW is algebraic, not restricted to DMDA. Here is an example command line. -pc_type mg -pc_mg_levels 2 -pc_mg_adapt_interp_coarse_space *gdsw* -pc_mg_galerkin -mg_levels_pc_type jacobi If you have a MATIS matrix, you can also turn on adaptivity (see for example these tests). https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/ksp/tutorials/ex71.c*L727__;Iw!!G_uCfscf7eWS!dU8Plji5xR1NEXhvUg83_jf1O061HWdtZYZjkpWxufOe3ZueuCJ7WMjvMquhjDVHAspyVJpaem8_0EkiWRiVIYI6IyiGlDw$ https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/snes/tutorials/ex12.c*L1799__;Iw!!G_uCfscf7eWS!dU8Plji5xR1NEXhvUg83_jf1O061HWdtZYZjkpWxufOe3ZueuCJ7WMjvMquhjDVHAspyVJpaem8_0EkiWRiVIYI6ndo6f0o$ Code setup is here https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/mg/gdsw.c__;!!G_uCfscf7eWS!dU8Plji5xR1NEXhvUg83_jf1O061HWdtZYZjkpWxufOe3ZueuCJ7WMjvMquhjDVHAspyVJpaem8_0EkiWRiVIYI6bWXW_Co$ Il giorno gio 22 mag 2025 alle ore 10:59 Edoardo Centofanti < edoardo.centofanti01 at universitadipavia.it> ha scritto: > Dear all, > > Thank you for your answers. > Do you know which options are required for a typical use case of GDSW in > PCMG? However, I imagine that only structured DMDA meshes can be employed > with it. > > Best, > Edoardo > > Il giorno mer 21 mag 2025 alle ore 16:52 Barry Smith > ha scritto: > >> >> Thanks for your interest in this type of algorithm. >> >> As Pierre notes, the code is implemented only as a simple tutorial on >> one particular "exotic" wire basket-type algorithm. We don't recommend this >> implementation for more general algorithms or geometries. >> >> The face dof represents some sort of "average" of the face solution, it >> is not a point value. >> >> Barry >> >> >> On May 21, 2025, at 4:51?AM, Pierre Jolivet wrote: >> >> >> >> On 21 May 2025, at 10:41?AM, Edoardo Centofanti < >> edoardo.centofanti01 at universitadipavia.it> wrote: >> >> ? >> Dear all, >> >> I have some questions about the PCEXOTIC preconditioner: >> >> 1- While the type "wirebasket" seems clear to me, I do not understand >> what is meant as "one dof per face" in the documentation referring to >> "face" type. Is it intended as the interpolation on the centre of the face >> or some mean value? >> >> 2- Is it possible to change the overlap between the subdomains? >> >> 3- Are there any plans of extending the implementation to unstructured >> grids (DMPlex-like)? The current one seems to rely heavily on DMDA >> >> 4- Running some preliminary tests on GPU with a code of mine that works >> on CPU, I got the following error: >> >> ** On entry to cusparseSpMM_bufferSize(): dimension mismatch, >> matA.num_rows (15405) != matC.num_rows (44955) with opA = >> CUSPARSE_OPERATION_NON_TRANSPOSE >> >> Is the preconditioner intended to run also on GPU or the error above >> comes from the fact that this feature has yet to be tested/fully >> implemented? >> >> >> This preconditioner is very old and the code is barely maintained. >> There is a newer implementation of the GDSW preconditioner in PCMG, see >> https://urldefense.us/v3/__https://petsc.org/release/manualpages/PC/PCMGSetAdaptCoarseSpaceType/__;!!G_uCfscf7eWS!dU8Plji5xR1NEXhvUg83_jf1O061HWdtZYZjkpWxufOe3ZueuCJ7WMjvMquhjDVHAspyVJpaem8_0EkiWRiVIYI68ia6R0o$ >> >> >> Thanks, >> Pierre >> >> Thank you in advance, >> Best, >> Edoardo >> >> >> > > -- > Edoardo Centofanti > Dipartimento di Matematica 'Felice Casorati' > Universit? degli Studi di Pavia > > Tel. 0382985608 | Ufficio A14 | Via Ferrata 5, 27100 Pavia, Italy > > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From edoardo.centofanti01 at universitadipavia.it Thu May 22 04:07:52 2025 From: edoardo.centofanti01 at universitadipavia.it (Edoardo Centofanti) Date: Thu, 22 May 2025 11:07:52 +0200 Subject: [petsc-users] Questions about PCEXOTIC In-Reply-To: References: <3C3F282B-8E28-4831-8955-14E122808C87@joliv.et> <2CF3E79F-09C2-4A77-AEC2-82B969DC6181@petsc.dev> Message-ID: Dear Stefano, Very nice, that was exactly what I was looking for. Also the possibility of turning on adaptivity with MATIS seems an interesting option. Best, Edoardo Il giorno gio 22 mag 2025 alle ore 10:33 Stefano Zampini < stefano.zampini at gmail.com> ha scritto: > GDSW is algebraic, not restricted to DMDA. Here is an example command > line. > > -pc_type mg -pc_mg_levels 2 -pc_mg_adapt_interp_coarse_space *gdsw* > -pc_mg_galerkin -mg_levels_pc_type jacobi > > > If you have a MATIS matrix, you can also turn on adaptivity (see for > example these tests). > > > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/ksp/tutorials/ex71.c*L727__;Iw!!G_uCfscf7eWS!Z06Y5N-tG2d-bTV_5UgqncaOVFzjv3GVertAowz01EIeSx2WIgHrNtv_gI26zxcuxrh6IaMuDaQe0Vx6FO3MYaMxOWXsQvfTFv9HGTisNCo$ > > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/snes/tutorials/ex12.c*L1799__;Iw!!G_uCfscf7eWS!Z06Y5N-tG2d-bTV_5UgqncaOVFzjv3GVertAowz01EIeSx2WIgHrNtv_gI26zxcuxrh6IaMuDaQe0Vx6FO3MYaMxOWXsQvfTFv9HS314Dc0$ > > > > Code setup is here > https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/mg/gdsw.c__;!!G_uCfscf7eWS!Z06Y5N-tG2d-bTV_5UgqncaOVFzjv3GVertAowz01EIeSx2WIgHrNtv_gI26zxcuxrh6IaMuDaQe0Vx6FO3MYaMxOWXsQvfTFv9H9bF7URM$ > > > > > Il giorno gio 22 mag 2025 alle ore 10:59 Edoardo Centofanti < > edoardo.centofanti01 at universitadipavia.it> ha scritto: > >> Dear all, >> >> Thank you for your answers. >> Do you know which options are required for a typical use case of GDSW in >> PCMG? However, I imagine that only structured DMDA meshes can be employed >> with it. >> >> Best, >> Edoardo >> >> Il giorno mer 21 mag 2025 alle ore 16:52 Barry Smith >> ha scritto: >> >>> >>> Thanks for your interest in this type of algorithm. >>> >>> As Pierre notes, the code is implemented only as a simple tutorial on >>> one particular "exotic" wire basket-type algorithm. We don't recommend this >>> implementation for more general algorithms or geometries. >>> >>> The face dof represents some sort of "average" of the face solution, >>> it is not a point value. >>> >>> Barry >>> >>> >>> On May 21, 2025, at 4:51?AM, Pierre Jolivet wrote: >>> >>> >>> >>> On 21 May 2025, at 10:41?AM, Edoardo Centofanti < >>> edoardo.centofanti01 at universitadipavia.it> wrote: >>> >>> ? >>> Dear all, >>> >>> I have some questions about the PCEXOTIC preconditioner: >>> >>> 1- While the type "wirebasket" seems clear to me, I do not understand >>> what is meant as "one dof per face" in the documentation referring to >>> "face" type. Is it intended as the interpolation on the centre of the face >>> or some mean value? >>> >>> 2- Is it possible to change the overlap between the subdomains? >>> >>> 3- Are there any plans of extending the implementation to unstructured >>> grids (DMPlex-like)? The current one seems to rely heavily on DMDA >>> >>> 4- Running some preliminary tests on GPU with a code of mine that works >>> on CPU, I got the following error: >>> >>> ** On entry to cusparseSpMM_bufferSize(): dimension mismatch, >>> matA.num_rows (15405) != matC.num_rows (44955) with opA = >>> CUSPARSE_OPERATION_NON_TRANSPOSE >>> >>> Is the preconditioner intended to run also on GPU or the error above >>> comes from the fact that this feature has yet to be tested/fully >>> implemented? >>> >>> >>> This preconditioner is very old and the code is barely maintained. >>> There is a newer implementation of the GDSW preconditioner in PCMG, see >>> https://urldefense.us/v3/__https://petsc.org/release/manualpages/PC/PCMGSetAdaptCoarseSpaceType/__;!!G_uCfscf7eWS!Z06Y5N-tG2d-bTV_5UgqncaOVFzjv3GVertAowz01EIeSx2WIgHrNtv_gI26zxcuxrh6IaMuDaQe0Vx6FO3MYaMxOWXsQvfTFv9HS4XKaQc$ >>> >>> >>> Thanks, >>> Pierre >>> >>> Thank you in advance, >>> Best, >>> Edoardo >>> >>> >>> >> >> -- >> Edoardo Centofanti >> Dipartimento di Matematica 'Felice Casorati' >> Universit? degli Studi di Pavia >> >> Tel. 0382985608 | Ufficio A14 | Via Ferrata 5, 27100 Pavia, Italy >> >> > > > -- > Stefano > -- Edoardo Centofanti Dipartimento di Matematica 'Felice Casorati' Universit? degli Studi di Pavia Tel. 0382985608 | Ufficio A14 | Via Ferrata 5, 27100 Pavia, Italy -------------- next part -------------- An HTML attachment was scrubbed... URL: From superdduck88 at gmail.com Thu May 22 01:23:55 2025 From: superdduck88 at gmail.com (Donald Duck) Date: Thu, 22 May 2025 08:23:55 +0200 Subject: [petsc-users] PETSc read binary matrix row by row Message-ID: Hello everyone A piece of c++ code writes PETSc AIJ matrices to binary files. My task is to compute an average matrix of these AIJ matrices. Therefore I read the first matrix with petsc4py and then start to add the other matrices to it. All matrixes always have the same size, shape, nnz etc. However in some cases these matrices are large and only one of it fits into memory and the reading/writing takes a significant amout of time. Is there a way to read it row by row to prevent memory overflow? I'm also open for other suggestions, thanks in advance! Raphael -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Thu May 22 09:51:51 2025 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 22 May 2025 09:51:51 -0500 Subject: [petsc-users] PETSc read binary matrix row by row In-Reply-To: References: Message-ID: Did you run in MPI parallel? If not, using MPI and running with multiple compute nodes could solve the problem. Are all these matrices already on disk? Then you have to pay the I/O cost for reading the matrices. --Junchao Zhang On Thu, May 22, 2025 at 8:21?AM Donald Duck wrote: > Hello everyone > > A piece of c++ code writes PETSc AIJ matrices to binary files. My task is > to compute an average matrix of these AIJ matrices. Therefore I read the > first matrix with petsc4py and then start to add the other matrices to it. > All matrixes always have the same size, shape, nnz etc. > > However in some cases these matrices are large and only one of it fits > into memory and the reading/writing takes a significant amout of time. Is > there a way to read it row by row to prevent memory overflow? > > I'm also open for other suggestions, thanks in advance! > > Raphael > -------------- next part -------------- An HTML attachment was scrubbed... URL: From superdduck88 at gmail.com Thu May 22 10:45:49 2025 From: superdduck88 at gmail.com (superdduck88 at gmail.com) Date: Thu, 22 May 2025 17:45:49 +0200 Subject: [petsc-users] PETSc read binary matrix row by row In-Reply-To: References: Message-ID: <161390A9-2734-4064-B65C-8BEFF298D817@gmail.com> An HTML attachment was scrubbed... URL: From dontbugthedevs at proton.me Thu May 22 11:25:48 2025 From: dontbugthedevs at proton.me (Noam T.) Date: Thu, 22 May 2025 16:25:48 +0000 Subject: [petsc-users] Element connectivity of a DMPlex In-Reply-To: References: Message-ID: Hello, Thank you the various options. Use case here would be obtaining the exact output generated by option 1), DMView() with PETSC_VIEWER_HDF5_VIZ; in particular, the matrix generated under /viz/topology/cells. > There are several ways you might do this. It helps to know what you are aiming for. > > 1) If you just want this output, it might be easier to just DMView() with the PETSC_VIEWER_HDF5_VIZ format, since that just outputs the cell-vertex topology and coordinates Is it possible to get this information in memory, onto a Mat, Vec or some other Int array object directly? it would be handy to have it in order to manipulate it and/or save it to a different format/file. Saving to an HDF5 and loading it again seems redundant. > 2) You can call DMPlexUninterpolate() to produce a mesh with just cells and vertices, and output it in any format. > > 3) If you want it in memory, but still with global indices (I don't understand this use case), then you can use DMPlexCreatePointNumbering() for an overall global numbering, or DMPlexCreateCellNumbering() and DMPlexCreateVertexNumbering() for separate global numberings. Perhaps I missed it, but getting the connectivity matrix in /viz/topology/cells/ did not seem directly trivial to me from the list of global indices returned by DMPlexGetCell/Point/VertexNumbering() (i.e. I assume all the operations done when calling DMView()). Thanks, Noam. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Thu May 22 11:54:33 2025 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Thu, 22 May 2025 19:54:33 +0300 Subject: [petsc-users] PETSc read binary matrix row by row In-Reply-To: <161390A9-2734-4064-B65C-8BEFF298D817@gmail.com> References: <161390A9-2734-4064-B65C-8BEFF298D817@gmail.com> Message-ID: How big are these matrices? The memory PETSc allocates is (8+4)*num_nnz + 4*num_rows bytes (in double precision) https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/mat/impls/aij/seq/aij.h*L47__;Iw!!G_uCfscf7eWS!eQLvMSRayHm9_oWVvaVFu0KZKnXMgFnQ06zPI0ciy6V4RLioHdXqliTOEFJQs8W8qb-y8v2-wVe3mKbGwBDucIVgolkrDHg$ Assuming num_nnz >> num_rows, 2.8 billion nonzeros fit 32GB of RAM. Anyway, if you say you use the same job for matrix write + load + averaging, then you are better off allocating a single matrix and perform averaging in the C++ code directly allocate_and_zero_a single matrix for i in range(number_of_matrices) add_values_to_the_matrix scale_matrix_for_average Il giorno gio 22 mag 2025 alle ore 18:55 superdduck88 at gmail.com < superdduck88 at gmail.com> ha scritto: > Hi > > The averaging takes place on the same hardware and as part of the same job > as the writing the matrix to file in the c++ code. The number of nodes is > determined on the problem size and therefor for a single matrix. Allocating > extra nodes for the averaging is not economical, unfortunately. > > On 22 May 2025, at 16:52, Junchao Zhang wrote: > > ? > Did you run in MPI parallel? If not, using MPI and running with multiple > compute nodes could solve the problem. > > Are all these matrices already on disk? Then you have to pay the I/O cost > for reading the matrices. > > --Junchao Zhang > > > On Thu, May 22, 2025 at 8:21?AM Donald Duck > wrote: > >> Hello everyone >> >> A piece of c++ code writes PETSc AIJ matrices to binary files. My task is >> to compute an average matrix of these AIJ matrices. Therefore I read the >> first matrix with petsc4py and then start to add the other matrices to it. >> All matrixes always have the same size, shape, nnz etc. >> >> However in some cases these matrices are large and only one of it fits >> into memory and the reading/writing takes a significant amout of time. Is >> there a way to read it row by row to prevent memory overflow? >> >> I'm also open for other suggestions, thanks in advance! >> >> Raphael >> > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu May 22 13:31:38 2025 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 22 May 2025 14:31:38 -0400 Subject: [petsc-users] PETSc read binary matrix row by row In-Reply-To: References: Message-ID: Take a look at MatLoad_SeqAIJ_Binary(). You will see how it is easy to simplify that code to get what you need. You can call PetscViewerBinaryGetDescriptor() and then use lseek() to skip over all the integer part of the matrix storage (the row lengths and column indices) and not even read that into memory at all. Since the file stores all the numerical values after the integer part you won't need to read a row at a time, just loop over "big chunks" and read a big chunk at a time of the numerical part and add that big chunk to the appropriate place in the in-memory AIJ matrix. You are free to pick any big chunk size, it should be at least 1000s of doubles for good performance. Barry > On May 22, 2025, at 2:23?AM, Donald Duck wrote: > > Hello everyone > > A piece of c++ code writes PETSc AIJ matrices to binary files. My task is to compute an average matrix of these AIJ matrices. Therefore I read the first matrix with petsc4py and then start to add the other matrices to it. All matrixes always have the same size, shape, nnz etc. > > However in some cases these matrices are large and only one of it fits into memory and the reading/writing takes a significant amout of time. Is there a way to read it row by row to prevent memory overflow? > > I'm also open for other suggestions, thanks in advance! > > Raphael -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu May 22 19:56:27 2025 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 22 May 2025 20:56:27 -0400 Subject: [petsc-users] Element connectivity of a DMPlex In-Reply-To: References: Message-ID: On Thu, May 22, 2025 at 12:25?PM Noam T. wrote: > Hello, > > Thank you the various options. > > Use case here would be obtaining the exact output generated by option 1), > DMView() with PETSC_VIEWER_HDF5_VIZ; in particular, the matrix generated > under /viz/topology/cells. > > There are several ways you might do this. It helps to know what you are > aiming for. > > 1) If you just want this output, it might be easier to just DMView() with > the PETSC_VIEWER_HDF5_VIZ format, since that just outputs the cell-vertex > topology and coordinates > > > Is it possible to get this information in memory, onto a Mat, Vec or some > other Int array object directly? it would be handy to have it in order to > manipulate it and/or save it to a different format/file. Saving to an HDF5 > and loading it again seems redundant. > > > 2) You can call DMPlexUninterpolate() to produce a mesh with just cells > and vertices, and output it in any format. > > 3) If you want it in memory, but still with global indices (I don't > understand this use case), then you can use DMPlexCreatePointNumbering() > for an overall global numbering, or DMPlexCreateCellNumbering() and > DMPlexCreateVertexNumbering() for separate global numberings. > > > Perhaps I missed it, but getting the connectivity matrix in > /viz/topology/cells/ did not seem directly trivial to me from the list of > global indices returned by DMPlexGetCell/Point/VertexNumbering() (i.e. I > assume all the operations done when calling DMView()). > Something like DMPlexGetHeightStratum(dm, 0, &cStart, &cEnd); DMPlexGetDepthStratum(dm, 0, &vStart, &vEnd); DMPlexGetVertexNumbering(dm, &globalVertexNumbers); ISGetIndices(globalVertexNumbers, &gv); for (PetscInt c = cStart; c < cEnd; ++c) { PetscInt *closure = NULL; DMPlexGetTransitiveClosure(dm, c, PETSC_TRUE, &Ncl, &closure); for (PetscInt cl = 0; c < Ncl * 2; cl += 2) { if (closure[cl] < vStart || closure[cl] >= vEnd) continue; const PetscInt v = gv[closure[cl]] < 0 ? -(gv[closure[cl]] + 1) : gv[closure[cl]]; // Do something with v } DMPlexRestoreTransitiveClosure(dm, c, PETSC_TRUE, &Ncl, &closure); } ISRestoreIndices(globalVertexNumbers, &gv); ISDestroy(&globalVertexNumbers); Thanks, Matt > Thanks, > Noam. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!c-kIsacyZtVK0QT5hSQ6TYjx-_vFWjrr6SXhV0Q8o9GcMLSLwko9mlT7CXjpNSVCT4bRPKeSkCoQJMg2SkFk$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yc17470 at connect.um.edu.mo Mon May 26 03:45:40 2025 From: yc17470 at connect.um.edu.mo (Gong Yujie) Date: Mon, 26 May 2025 08:45:40 +0000 Subject: [petsc-users] Questions about PetscTabulation for finite element basis, and PetscViewerVTK for saving cell data Message-ID: Dear PETSc developers, I have a code that uses PETSc to solve the elasticity problem with the finite element discretization written by myself. I found there is a part called "PetscTabulation" that stores the basis functions. I would like to inquire about this part that can I use this part for achieving high order element? In detail, how does the basis functions be evaluated and in which file can I find the basis functions? I'm thinking about using PetscTabulation and PetscQuadrature to achieve a high order element. Is this doable? The next question is can I output a vector that the unknown is defined on each element (not point) using VecView? I think this question relates to the PetscViewer for VTK. Thanks in advance for your help! Best Regards, Jerry -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon May 26 09:49:14 2025 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 26 May 2025 10:49:14 -0400 Subject: [petsc-users] Questions about PetscTabulation for finite element basis, and PetscViewerVTK for saving cell data In-Reply-To: References: Message-ID: On Mon, May 26, 2025 at 4:45?AM Gong Yujie wrote: > Dear PETSc developers, > > I have a code that uses PETSc to solve the elasticity problem with the > finite element discretization written by myself. I found there is a part > called "PetscTabulation" that stores the basis functions. I would like to > inquire about this part that can I use this part for achieving high order > element? In detail, how does the basis functions be evaluated and in which > file can I find the basis functions? I'm thinking about using > PetscTabulation and PetscQuadrature to achieve a high order element. Is > this doable? > PetscTabulation just holds the evaluation of the basis functions at the quadrature points (or any other points). PetscQuadrature generates quadrature points on polytopes, with some quadrature order. You can use them independently of PetscFE to do this job. > The next question is can I output a vector that the unknown is defined on > each element (not point) using VecView? I think this question relates to > the PetscViewer for VTK. > I have never gotten higher order elements to work in VTK. What I do is refine the mesh and output on cells or vertices. Thanks, Matt > Thanks in advance for your help! > > Best Regards, > Jerry > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!b-Pl5dk7k-iMvh1tKmqgN9CcjahtP03NNI-r20KFMR79FkgUlC6SUyA7mGDiK2_D_DO_Bb_7hRafxmCsIW5m$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ctchengben at mail.scut.edu.cn Wed May 28 11:36:47 2025 From: ctchengben at mail.scut.edu.cn (=?UTF-8?B?56iL5aWU?=) Date: Thu, 29 May 2025 00:36:47 +0800 (GMT+08:00) Subject: [petsc-users] Can PETSc support parallelly import HDF5 mesh file? Message-ID: <3541d2bc.29cf.19717c0e7f3.Coremail.ctchengben@mail.scut.edu.cn> Hello, Recently I create unstructure mesh from Gmsh and its mesh format is msh file. However the mesh file contain around 100 million nodes, so when I use DMPlexCreateFromFile it only perform on a single CPU process thus out of memory. I have sent email to report this situation to your guys and you told me that I can use h5 mesh file. Thanks for your advise so I try to generate h5 file. In order to load the very large msh, I first use a single computer node(64 CPU process) that with large CPU memory. I just use one CPU process to perform the code to generate h5 mesh file: ----------------------------------------------------------------------------------------------- //Set up or input the mesh ierr = DMPlexCreateFromFile(PETSC_COMM_WORLD, "mesh.msh","hjtest" ,PETSC_TRUE, &dm_nodes);CHKERRQ(ierr); //Distribute the mesh into different processers DM dm_nodes_Dist; //the dm object using to contain the distributed mesh temporarily PetscPartitioner part; //for distributing the mesh ierr = DMPlexGetPartitioner(dm_nodes, &part);CHKERRQ(ierr); //Get the partitioner of the mesh we just created ierr = PetscPartitionerSetType(part,PETSCPARTITIONERPARMETIS);CHKERRQ(ierr); //Set the partitioner from the commond line option ierr = DMPlexDistribute(dm_nodes,0,NULL,&dm_nodes_Dist); CHKERRQ(ierr); //distribute the mesh into different processers if (dm_nodes_Dist) {DMDestroy(&dm_nodes); dm_nodes = dm_nodes_Dist;} //delete the origin mesh and use the distributed one PetscViewerHDF5Open(PETSC_COMM_WORLD, "mesh.h5", FILE_MODE_WRITE, &viewer); DMView(dm_nodes, viewer); PetscViewerDestroy(&viewer); DMDestroy(&dm_nodes); ----------------------------------------------------------------------------------------------- I get the large h5 mesh, then I want to use another computer(equiped with 20 computer node 1280 CPU process that have not so large CPU memory) to perform the parallel computation. So I perform the code that use 20 computer node 1280 CPU process: ************************************************************************************************* ierr = PetscViewerHDF5Open(PETSC_COMM_WORLD, "mesh.h5", FILE_MODE_READ, &viewer); CHKERRQ(ierr); ierr = PetscViewerSetFormat(viewer, PETSC_VIEWER_HDF5); CHKERRQ(ierr); ierr = DMPlexCreate(PETSC_COMM_WORLD, &dm_nodes); CHKERRQ(ierr); ierr = DMSetType(dm_nodes, DMPLEX); CHKERRQ(ierr); PetscPartitioner part; ierr = DMPlexGetPartitioner(dm_nodes, &part); CHKERRQ(ierr); ierr = PetscPartitionerSetType(part, PETSCPARTITIONERPARMETIS); CHKERRQ(ierr); ierr = DMLoad(dm_nodes, viewer); CHKERRQ(ierr); ierr = PetscViewerDestroy(&viewer); CHKERRQ(ierr); PetscPrintf(PETSC_COMM_WORLD,"# Loaded mesh from HDF5\n"); ************************************************************************************************* I don't know if in this way, the PETSc can load h5 mesh file parallelly so that it will not only perform on a single CPU process thus out of memory. I try several times but it seems that it always load the h5 file on the one CPU process, I don't know if it have some mistakes or something. And I use the version of PETSc is 3.21.6 and the configure is attached, I do download the hdf5. So I write this email to ask for the help. Looking forward to your reply! sinserely, Cheng. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: configure.log URL: From knepley at gmail.com Wed May 28 11:56:03 2025 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 28 May 2025 12:56:03 -0400 Subject: [petsc-users] Can PETSc support parallelly import HDF5 mesh file? In-Reply-To: <3541d2bc.29cf.19717c0e7f3.Coremail.ctchengben@mail.scut.edu.cn> References: <3541d2bc.29cf.19717c0e7f3.Coremail.ctchengben@mail.scut.edu.cn> Message-ID: On Wed, May 28, 2025 at 12:38?PM ?? via petsc-users wrote: > Hello, > > Recently I create unstructure mesh from Gmsh and its mesh format is msh > file. However the mesh file contain around 100 million nodes, so when I use *DMPlexCreateFromFile > * > it only perform on a single CPU process thus out of memory. > > I have sent email to report this situation to your guys and you told me > that I can use h5 mesh file. Thanks for your advise so I try to generate h5 > file. > > In order to load the very large msh, I first use a single computer node(64 CPU > process) that with large CPU memory. I just use one CPU process to > perform the code to generate h5 mesh file: > > > ----------------------------------------------------------------------------------------------- > > > //Set up or input the mesh > > ierr = DMPlexCreateFromFile(PETSC_COMM_WORLD, "mesh.msh","hjtest" > ,PETSC_TRUE, &dm_nodes);CHKERRQ(ierr); > > > //Distribute the mesh into different processers > DM dm_nodes_Dist; //the dm object using to contain the distributed mesh > temporarily > PetscPartitioner part; //for distributing the mesh > ierr = DMPlexGetPartitioner(dm_nodes, &part);CHKERRQ(ierr); //Get the > partitioner of the mesh we just created > ierr = > PetscPartitionerSetType(part,PETSCPARTITIONERPARMETIS);CHKERRQ(ierr); //Set > the partitioner from the commond line option > ierr = DMPlexDistribute(dm_nodes,0,NULL,&dm_nodes_Dist); CHKERRQ(ierr); > //distribute the mesh into different processers > if (dm_nodes_Dist) {DMDestroy(&dm_nodes); dm_nodes = dm_nodes_Dist;} > //delete the origin mesh and use the distributed one > This distribution is unnecessary. > > PetscViewerHDF5Open(PETSC_COMM_WORLD, "mesh.h5", FILE_MODE_WRITE, &viewer); > DMView(dm_nodes, viewer); > PetscViewerDestroy(&viewer); > It would be easier to use PetscCall(DMViewFromOptions(dm_nodes, NULL, "-dm_view")); so that we could easily customize things. Here you will need to write the file in an updated format -dm_view hdf5:mesh.h5 -dm_plex_view_hdf5_storage_version 3.1.0 > > DMDestroy(&dm_nodes); > > ----------------------------------------------------------------------------------------------- > > > I get the large h5 mesh, then I want to use another computer(equiped with > 20 computer node 1280 CPU process that have not so large CPU memory) to > perform the parallel computation. So I perform the code that use 20 > computer node 1280 CPU process: > > > > ************************************************************************************************* > > ierr = PetscViewerHDF5Open(PETSC_COMM_WORLD, "mesh.h5", FILE_MODE_READ, > &viewer); CHKERRQ(ierr); > ierr = PetscViewerSetFormat(viewer, PETSC_VIEWER_HDF5); CHKERRQ(ierr); > > > > ierr = DMPlexCreate(PETSC_COMM_WORLD, &dm_nodes); CHKERRQ(ierr); > ierr = DMSetType(dm_nodes, DMPLEX); CHKERRQ(ierr); > > > > > PetscPartitioner part; > ierr = DMPlexGetPartitioner(dm_nodes, &part); CHKERRQ(ierr); > ierr = PetscPartitionerSetType(part, PETSCPARTITIONERPARMETIS); > CHKERRQ(ierr); > > > ierr = DMLoad(dm_nodes, viewer); CHKERRQ(ierr); > ierr = PetscViewerDestroy(&viewer); CHKERRQ(ierr); > I think it is simpler to just use PetscCall(DMCreate(comm, &dm)); PetscCall(DMSetType(dm, DMPLEX)); PetscCall(DMSetFromOptions(dm)); and then use -dm_plex_filename mesh.h5 to load it. Thanks, Matt > PetscPrintf(PETSC_COMM_WORLD,"# Loaded mesh from HDF5\n"); > > > ************************************************************************************************* > > > > I don't know if in this way, the PETSc can load h5 mesh file parallelly so > that it will not only perform on a single CPU process thus out of memory. > I try several times but it seems that it always load the h5 file on the one > CPU process, I don't know if it have some mistakes or something. > > > And I use the version of PETSc is 3.21.6 and the configure is attached, I > do download the hdf5. > > > So I write this email to ask for the help. > > > Looking forward to your reply! > > sinserely, > Cheng. > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YotTbyod-NJW43FaXi_Oc9Cqlb1kMpzfZv43sF-b6wDFrCR7USjxhBX_7BAwH2hQ-QuGQXxSBrkUJ67YeQ7C$ -------------- next part -------------- An HTML attachment was scrubbed... URL: