[petsc-users] problem with nested logging
Stefano Zampini
stefano.zampini at gmail.com
Thu May 1 03:57:08 CDT 2025
Il giorno gio 1 mag 2025 alle ore 11:38 Klaij, Christiaan <C.Klaij at marin.nl>
ha scritto:
> The checks seem to be in place: I do get a PETSC ERROR when I add a log
> event on rank 0 as you suggested.
>
>
Ok, the broken logic may be in LogView then. You can try deactivating some
logging by classes and see how the error evolves, maybe using
PetscLogClassSetActiveAll. Or, if feasible, commenting out some part of the
simulation code
> Another thought: in between the log events pairs, I also have calls to
> PetscLogFlops, perhaps that plays a role somehow.
>
> It shouldn't
> Chris
>
> ________________________________________
> From: Klaij, Christiaan <C.Klaij at marin.nl>
> Sent: Thursday, May 1, 2025 10:23 AM
> To: Stefano Zampini
> Cc: Randall Mackie; PETSc users list; Isaac, Toby
> Subject: Re: [petsc-users] problem with nested logging
>
> Was the rewritting by Toby done somewhere between petsc 3.19.4 (no
> problem) and 3.23.4 (problem)?
>
> Chris
>
> ________________________________________
> From: Stefano Zampini <stefano.zampini at gmail.com>
> Sent: Thursday, May 1, 2025 9:12 AM
> To: Klaij, Christiaan
> Cc: Randall Mackie; PETSc users list; Isaac, Toby
> Subject: Re: [petsc-users] problem with nested logging
>
> If I look at the code for PetscLogHandlerEventBegin_Default, there are
> checks in place to see if the event is collectively called (see below)
> Can you make sure the Nested logging system has the same checks?
> It should, but double check since the code has been largely rewritten by
> Toby some time ago; to check it should be as easy as writing a code that
> calls a collective event on a single process and a debug version of petsc
> should complain
>
> if (rank ==0)
> LogEventBegin() <-this should call MPIU_Allreduce, but other processes
> will not, thus hang
>
>
> If it does not complain, then the error must come from how the logic in
> LogView works, and from how it traverses the various events (my guess:
> processed in a different order from different processes). Without a
> reproducer, it is hard to understand what's going on
>
> static PetscErrorCode PetscLogHandlerEventBegin_Default(PetscLogHandler h,
> PetscLogEvent event, PetscObject o1, PetscObject o2, PetscObject o3,
> PetscObject o4)
> {
> PetscLogHandler_Default def =
> (PetscLogHandler_Default)h->data;
> PetscEventPerfInfo *event_perf_info = NULL;
> PetscLogEventInfo event_info;
> PetscLogDouble time;
> PetscLogState state;
> PetscLogStage stage;
>
> PetscFunctionBegin;
> PetscCall(PetscLogHandlerGetState(h, &state));
> if (PetscDefined(USE_DEBUG)) {
> PetscCall(PetscLogStateEventGetInfo(state, event, &event_info));
> if (PetscUnlikely(o1)) PetscValidHeader(o1, 3);
> if (PetscUnlikely(o2)) PetscValidHeader(o2, 4);
> if (PetscUnlikely(o3)) PetscValidHeader(o3, 5);
> if (PetscUnlikely(o4)) PetscValidHeader(o4, 6);
> if (event_info.collective && o1) {
> PetscInt64 b1[2], b2[2];
>
> b1[0] = -o1->cidx;
> b1[1] = o1->cidx;
> PetscCallMPI(MPIU_Allreduce(b1, b2, 2, MPIU_INT64, MPI_MAX,
> PetscObjectComm(o1)));
> PetscCheck(-b2[0] == b2[1], PETSC_COMM_SELF, PETSC_ERR_PLIB,
> "Collective event %s not called collectively %" PetscInt64_FMT " != %"
> PetscInt64_FMT, event_info.name<https://urldefense.us/v3/__http://event_info.name__;!!G_uCfscf7eWS!crCM5sjjbiW2UA3oXM3tEMAALyn_2tPaKI8PGqZHWnSyRxkWGSkyxCXMSOOX4ahR0lDueyvZM3jiyD8WwsnM8kQGIzRjQos$ >, -b2[0], b2[1]);
> }
> }
> /* Synchronization */
> PetscCall(PetscLogHandlerEventSync_Default(h, event,
> PetscObjectComm(o1)));
>
>
>
>
> Il giorno gio 1 mag 2025 alle ore 09:56 Klaij, Christiaan <
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>> ha scritto:
> I've tried with -log_sync, no complaints whatsoever, but the error is
> still there...
>
> Chris
>
> ________________________________________
> From: Stefano Zampini <stefano.zampini at gmail.com<mailto:
> stefano.zampini at gmail.com>>
> Sent: Tuesday, April 29, 2025 6:12 PM
> To: Randall Mackie
> Cc: Klaij, Christiaan; PETSc users list; Isaac, Toby
> Subject: Re: [petsc-users] problem with nested logging
>
> Can you try using -log_sync ? This should check every entry/exit points of
> logged Events and complain if something is not collectively called
>
> Stefano
>
> On Tue, Apr 29, 2025, 18:21 Randall Mackie <rlmackie862 at gmail.com<mailto:
> rlmackie862 at gmail.com><mailto:rlmackie862 at gmail.com<mailto:
> rlmackie862 at gmail.com>>> wrote:
> ah okay, I missed that this was found using openmpi.
>
> then it’s probably not the same issue we had.
>
> I can’t remember in which version it was fixed (I’m away from my work
> computer)….I do know in our case openmpi and the latest Intel One API work
> fine.
>
> Randy
>
> On Apr 29, 2025, at 8:58 AM, Klaij, Christiaan <C.Klaij at marin.nl<mailto:
> C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>>
> wrote:
>
> Well, the error below only shows-up thanks to openmpi and gnu compilers.
> With the intel mpi and compilers it just hangs (tried oneapi 2023.1.0). In
> which version was that bug fixed?
>
> Chris
>
> ________________________________________
> <image374583.png>
> dr. ir. Christiaan Klaij
> | senior researcher | Research & Development |
> CFD Development
> T +31 317 49 33 44<tel:+31%20317%2049%2033%2044> |
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:
> C.Klaij at marin.nl>> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!crCM5sjjbiW2UA3oXM3tEMAALyn_2tPaKI8PGqZHWnSyRxkWGSkyxCXMSOOX4ahR0lDueyvZM3jiyD8WwsnM8kQGA8tjmMQ$ <https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!crCM5sjjbiW2UA3oXM3tEMAALyn_2tPaKI8PGqZHWnSyRxkWGSkyxCXMSOOX4ahR0lDueyvZM3jiyD8WwsnM8kQGA8tjmMQ$ ><
> https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cx-ci9iNg$
> >
> <image322757.png><
> https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cxl-rBbfA$
> >
> <image512282.png><
> https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cw7FNelIQ$
> >
> <image090879.png><
> https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6CwmfHQRog$
> >
>
> From: Randall Mackie <rlmackie862 at gmail.com<mailto:rlmackie862 at gmail.com
> ><mailto:rlmackie862 at gmail.com<mailto:rlmackie862 at gmail.com>>>
> Sent: Tuesday, April 29, 2025 3:33 PM
> To: Klaij, Christiaan
> Cc: Matthew Knepley; petsc-users at mcs.anl.gov<mailto:
> petsc-users at mcs.anl.gov><mailto:petsc-users at mcs.anl.gov<mailto:
> petsc-users at mcs.anl.gov>>; Isaac, Toby
> Subject: Re: [petsc-users] problem with nested logging
>
> You don't often get email from rlmackie862 at gmail.com<mailto:
> rlmackie862 at gmail.com><mailto:rlmackie862 at gmail.com<mailto:
> rlmackie862 at gmail.com>>. Learn why this is important<
> https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!crCM5sjjbiW2UA3oXM3tEMAALyn_2tPaKI8PGqZHWnSyRxkWGSkyxCXMSOOX4ahR0lDueyvZM3jiyD8WwsnM8kQG_JZIR7c$ <
> https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6CwgJDUS5w$
> >>
> We had a similar issue last year that we eventually tracked down to a bug
> in Intel MPI AllReduce, which was around the same version you are using.
>
> Can you try a different MPI or the latest Intel One API and see if your
> error clears?
>
> Randy
>
> On Tue, Apr 29, 2025 at 8:17 AM Klaij, Christiaan via petsc-users <
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>>> wrote:
> I don't think so, we have tracing in place to detect mismatches. But as
> soon as I switch the tracing on, the error disappears... Same if I add a
> counter or print statements before and after EventBegin/End. Looks like a
> memory corruption problem, maybe nothing to do with petsc despite the error
> message.
>
> Chris
>
> ________________________________________
> From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com><mailto:
> knepley at gmail.com<mailto:knepley at gmail.com>><mailto:knepley at gmail.com
> <mailto:knepley at gmail.com><mailto:knepley at gmail.com<mailto:
> knepley at gmail.com>>>>
> Sent: Tuesday, April 29, 2025 1:50 PM
> To: Klaij, Christiaan
> Cc: Junchao Zhang; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov
> ><mailto:petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>>; Isaac, Toby
> Subject: Re: [petsc-users] problem with nested logging
>
> On Tue, Apr 29, 2025 at 6:50 AM Klaij, Christiaan <C.Klaij at marin.nl
> <mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>><mailto:C.Klaij at marin.nl
> <mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>>>> wrote:
> Here's a slightly better error message, obtained --with-debugging=1
>
> Is it possible that you have a mismatched EventBegin()/EventEnd() in your
> code? That could be why we cannot reproduce it here.
>
> Thanks,
>
> Matt
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: MPIU_Allreduce() called in different locations (code
> lines) on different processors
> [0]PETSC ERROR: See
> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjggvQAzPU$
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025
> [0]PETSC ERROR: ./refresco with 2 MPI process(es) and PETSC_ARCH on
> marclus3login2 by cklaij Tue Apr 29 12:43:54 2025
> [0]PETSC ERROR: Configure options:
> --prefix=/home/cklaij/ReFRESCO/trunk/install/extLibs
> --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0
> --with-debugging=1 --download-superlu_dist=
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgVgVAJPM$
> --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0
> --download-parmetis=
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgo2JWTO4$
> --download-metis=
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgX9ZMYJA$
> --with-packages-build-dir=/home/cklaij/ReFRESCO/trunk/build-libs/superbuild
> --with-ssl=0 --with-shared-libraries=1
> [0]PETSC ERROR: #1 PetscLogNestedTreePrintLine() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:289
> [0]PETSC ERROR: #2 PetscLogNestedTreePrint() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:379
> [0]PETSC ERROR: #3 PetscLogNestedTreePrint() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384
> [0]PETSC ERROR: #4 PetscLogNestedTreePrint() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384
> [0]PETSC ERROR: #5 PetscLogNestedTreePrintTop() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420
> [0]PETSC ERROR: #6 PetscLogHandlerView_Nested_XML() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443
> [0]PETSC ERROR: #7 PetscLogHandlerView_Nested() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405
> [0]PETSC ERROR: #8 PetscLogHandlerView() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342
> [0]PETSC ERROR: #9 PetscLogView() at
> /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/plog.c:2040
> [0]PETSC ERROR: #10
> /home/cklaij/ReFRESCO/trunk/Code/src/petsc_include_impl.F90:130
>
> ________________________________________
> [cid:ii_19681617e7812ff9cfc1]
> dr. ir. Christiaan Klaij
> | senior researcher | Research & Development | CFD Development
> T +31 317 49 33 44<tel:+31%20317%2049%2033%2044> | C.Klaij at marin.nl
> <mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>><mailto:C.Klaij at marin.nl
> <mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>>> |
> https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgJooxJhg$
> <
> https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgpOXjQSM$
> >
> [Facebook]<
> https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg_3Zt0Pw$
> >
> [LinkedIn]<
> https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgcpgQdSE$
> >
> [YouTube]<
> https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgn8qP6_I$
> >
>
> From: Klaij, Christiaan <C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>><mailto:C.Klaij at marin.nl<mailto:
> C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >>><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>><mailto:C.Klaij at marin.nl<mailto:
> C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>>>>
> Sent: Monday, April 28, 2025 3:53 PM
> To: Matthew Knepley
> Cc: Junchao Zhang; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov
> ><mailto:petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>>>; Isaac, Toby
> Subject: Re: [petsc-users] problem with nested logging
>
> Bisecting would be quite hard, it's not just the petsc version that
> changed, also other libs, compilers, even os components.
>
> Chris
>
> ________________________________________
> From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com><mailto:
> knepley at gmail.com<mailto:knepley at gmail.com>><mailto:knepley at gmail.com
> <mailto:knepley at gmail.com><mailto:knepley at gmail.com<mailto:
> knepley at gmail.com>>><mailto:knepley at gmail.com<mailto:knepley at gmail.com
> ><mailto:knepley at gmail.com<mailto:knepley at gmail.com>><mailto:
> knepley at gmail.com<mailto:knepley at gmail.com><mailto:knepley at gmail.com
> <mailto:knepley at gmail.com>>>>>
> Sent: Monday, April 28, 2025 3:06 PM
> To: Klaij, Christiaan
> Cc: Junchao Zhang; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov
> ><mailto:petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>>>; Isaac, Toby
> Subject: Re: [petsc-users] problem with nested logging
>
> You don't often get email from knepley at gmail.com<mailto:knepley at gmail.com
> ><mailto:knepley at gmail.com<mailto:knepley at gmail.com>><mailto:
> knepley at gmail.com<mailto:knepley at gmail.com><mailto:knepley at gmail.com
> <mailto:knepley at gmail.com>>><mailto:knepley at gmail.com<mailto:
> knepley at gmail.com><mailto:knepley at gmail.com<mailto:knepley at gmail.com
> >><mailto:knepley at gmail.com<mailto:knepley at gmail.com><mailto:
> knepley at gmail.com<mailto:knepley at gmail.com>>>>. Learn why this is
> important<
> https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgcKEJXRA$
> >
> On Mon, Apr 28, 2025 at 8:45 AM Klaij, Christiaan via petsc-users <
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>>>>> wrote:
> I've tried adding a nested log viewer to src/snes/tutorials/ex70.c,
> but it does not replicate the problem and works fine.
>
> Perhaps it is related to fortran, since the manualpage of
> PetscLogNestedBegin says "No fortran support" (why? we've been
> using it in fortran ever since).
>
> Therefore I've tried adding it to src/snes/ex5f90.F90 and that
> also works fine. It seems I cannot replicate the problem in a
> small example, unfortunately.
>
> We cannot replicate it here. Is there a chance you could bisect to see
> what change is responsible?
>
> Thanks,
>
> Matt
>
> Chris
>
> ________________________________________
> From: Junchao Zhang <junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>>>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>>>>>>
> Sent: Saturday, April 26, 2025 3:51 PM
> To: Klaij, Christiaan
> Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>>>>; Isaac, Toby
> Subject: Re: [petsc-users] problem with nested logging
>
> You don't often get email from junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>>>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com><mailto:junchao.zhang at gmail.com<mailto:
> junchao.zhang at gmail.com>>>>>. Learn why this is important<
> https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gVgt1dmE$
> >
> Toby (Cc'ed) might know it. Or could you provide an example?
>
> --Junchao Zhang
>
>
> On Fri, Apr 25, 2025 at 3:31 AM Klaij, Christiaan via petsc-users <
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov><mailto:
> petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>>>>>> wrote:
> We recently upgraded from 3.19.4 to 3.22.4 but face the problem below with
> the nested logging. Any ideas?
>
> Chris
>
>
> [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: General MPI error
> [1]PETSC ERROR: MPI error 1 MPI_ERR_BUFFER: invalid buffer pointer
> [1]PETSC ERROR: See
> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gIT68pbk$
> <
> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM3UaqHzc$>
> for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025
> [1]PETSC ERROR: refresco with 2 MPI process(es) and PETSC_ARCH on
> marclus3login2 by jwindt Fri Apr 25 08:52:30 2025
> [1]PETSC ERROR: Configure options:
> --prefix=/home/jwindt/cmake_builds/refresco/install-libs-gnu
> --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0
> --with-debugging=0 --download-superlu_dist=
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6grH5BbeU$
> <
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM21-2D-o$>
> --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0
> --download-parmetis=
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gw4-tEtY$
> <
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMW0lYHko$>
> --download-metis=
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gHq4uYiY$
> <
> https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMbSrIiUg$>
> --with-packages-build-dir=/home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild
> --with-ssl=0 --with-shared-libraries=1 CFLAGS="-std=gnu11 -Wall
> -funroll-all-loops -O3 -DNDEBUG" CXXFLAGS="-std=gnu++14 -Wall
> -funroll-all-loops -O3 -DNDEBUG " COPTFLAGS="-std=gnu11 -Wall
> -funroll-all-loops -O3 -DNDEBUG" CXXOPTFLAGS="-std=gnu++14 -Wall
> -funroll-all-loops -O3 -DNDEBUG " FCFLAGS="-Wall -funroll-all-loops
> -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime
> -Wno-unused-function -O3 -DNDEBUG" F90FLAGS="-Wall -funroll-all-loops
> -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime
> -Wno-unused-function -O3 -DNDEBUG" FOPTFLAGS="-Wall -funroll-all-loops
> -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime
> -Wno-unused-function -O3 -DNDEBUG"
> [1]PETSC ERROR: #1 PetscLogNestedTreePrint() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:330
> [1]PETSC ERROR: #2 PetscLogNestedTreePrint() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384
> [1]PETSC ERROR: #3 PetscLogNestedTreePrint() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384
> [1]PETSC ERROR: #4 PetscLogNestedTreePrintTop() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420
> [1]PETSC ERROR: #5 PetscLogHandlerView_Nested_XML() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443
> [1]PETSC ERROR: #6 PetscLogHandlerView_Nested() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405
> [1]PETSC ERROR: #7 PetscLogHandlerView() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342
> [1]PETSC ERROR: #8 PetscLogView() at
> /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/plog.c:2040
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
> with errorcode 98.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> [cid:ii_196725d1e2a809852191]
> dr. ir. Christiaan Klaij
> | senior researcher | Research & Development | CFD Development
> T +31 317 49 33 44<tel:+31%20317%2049%2033%2044> | C.Klaij at marin.nl
> <mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>><mailto:C.Klaij at marin.nl
> <mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>>><mailto:C.Klaij at marin.nl
> <mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>><mailto:C.Klaij at marin.nl
> <mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>>>><mailto:C.Klaij at marin.nl
> <mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>><mailto:C.Klaij at marin.nl
> <mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>>><mailto:C.Klaij at marin.nl
> <mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>><mailto:C.Klaij at marin.nl
> <mailto:C.Klaij at marin.nl><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl
> >><mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl><mailto:
> C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>>>>>> |
> https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6g8TwMMcw$
> <
> https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM8tSNH1g$
> >
> [Facebook]<
> https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMcVCZ9hk$
> >
> [LinkedIn]<
> https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMIDBZW7k$
> >
> [YouTube]<
> https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMVKWos24$
> >
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$
> <
> https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgMsu6hhA$
> >
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$
> <
> https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgMsu6hhA$
> >
>
>
>
> --
> Stefano
>
--
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250501/ef794e17/attachment-0001.html>
More information about the petsc-users
mailing list