[petsc-users] problem with nested logging, standalone example

Thu Jul 10 03:15:05 CDT 2025

Hi Junchao,

Thanks for testing. I've fixed the error but unfortunately that doesn't change the behavior, the code still hangs as before, with the same stack trace...

Chris

_____
dr. ir. Christiaan Klaij | senior researcher
Research & Development | CFD Development
T +31 317 49 33 44 | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!bV-wSzFdop4YFdAYdaN5SqxF1Ea1onC_aTyxTwYg3TAYhYJpnIuO7MDbEv5VmkfbhJpB3EHweorcvDLsr_38Wj4$ 
___________________________________
From: Junchao Zhang <junchao.zhang at gmail.com>
Sent: Tuesday, July 8, 2025 10:58 PM
To: Klaij, Christiaan
Cc: PETSc users list
Subject: Re: [petsc-users] problem with nested logging, standalone example

Hi, Chris,
  First, I had to fix an error in your test by adding " PetscCallA(MatSetFromOptions(AA,ierr))" at line 254.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Mat object's type is not set: Argument # 1
...
[0]PETSC ERROR: #1 MatSetValues() at /scratch/jczhang/petsc/src/mat/interface/matrix.c:1503
[0]PETSC ERROR: #2 ex2f.F90:258

Then I could ran the test without problems
mpirun  -n 2 ./ex2f -pc_type jacobi -ksp_monitor_short -ksp_gmres_cgs_refinement_type refine_always
  0 KSP Residual norm 1.11803
  1 KSP Residual norm 0.591608
  2 KSP Residual norm 0.316228
  3 KSP Residual norm < 1.e-11
  0 KSP Residual norm 0.707107
  1 KSP Residual norm 0.408248
  2 KSP Residual norm < 1.e-11
Norm of error < 1.e-12 iterations     3

I used petsc-3.22.4, gcc-11.3, openmpi-5.0.6 and configured with
./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-openmpi --with-ssl=0 --with-shared-libraries=1 CFLAGS="-std=gnu11 -Wall -funroll-all-loops  -O3 -DNDEBUG" CXXFLAGS="-std=gnu++14 -Wall -funroll-all-loops  -O3 -DNDEBUG " COPTFLAGS="-std=gnu11 -Wall -funroll-all-loops  -O3 -DNDEBUG" CXXOPTFLAGS="-std=gnu++14 -Wall -funroll-all-loops  -O3 -DNDEBUG " FCFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" F90FLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" FOPTFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG"

Could you fix the error and retry?

--Junchao Zhang

On Sun, Jul 6, 2025 at 12:57 PM Klaij, Christiaan via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Attached is a standalone example of the issue described in the
earlier thread "problem with nested logging". The issue appeared
somewhere between petsc 3.19.4 and 3.23.4.

The example is a variation of ../ksp/tutorials/ex2f.F90, where
I've added the nested log viewer with one event as well as the
solution of a small system on rank zero.

When running on mulitple procs the example hangs during
PetscLogView with the backtrace below. The configure.log is also
attached in the hope that you can replicate the issue.

Chris

#0 0x000015554c84ea9e in mca_pml_ucx_recv (buf=0x7fffffff9e30, count=1,
datatype=0x15554c9ef900 <ompi_mpi_2dblprec>, src=1, tag=-12,
comm=0x7f1e30, mpi_status=0x0) at pml_ucx.c:700
#1 0x000015554c65baff in ompi_coll_base_allreduce_intra_recursivedoubling (
sbuf=0x7fffffff9e20, rbuf=0x7fffffff9e30, count=1,
dtype=0x15554c9ef900 <ompi_mpi_2dblprec>,
op=0x15554ca28980 <ompi_mpi_op_maxloc>, comm=0x7f1e30, module=0xaec630)
at base/coll_base_allreduce.c:247
#2 0x000015554c6a7e40 in ompi_coll_tuned_allreduce_intra_do_this (
sbuf=0x7fffffff9e20, rbuf=0x7fffffff9e30, count=1,
dtype=0x15554c9ef900 <ompi_mpi_2dblprec>,
op=0x15554ca28980 <ompi_mpi_op_maxloc>, comm=0x7f1e30, module=0xaec630,
algorithm=3, faninout=0, segsize=0) at coll_tuned_allreduce_decision.c:142
#3 0x000015554c6a054f in ompi_coll_tuned_allreduce_intra_dec_fixed (
sbuf=0x7fffffff9e20, rbuf=0x7fffffff9e30, count=1,
dtype=0x15554c9ef900 <ompi_mpi_2dblprec>,
op=0x15554ca28980 <ompi_mpi_op_maxloc>, comm=0x7f1e30, module=0xaec630)
at coll_tuned_decision_fixed.c:216
#4 0x000015554c68e160 in mca_coll_hcoll_allreduce (sbuf=0x7fffffff9e20,
rbuf=0x7fffffff9e30, count=1, dtype=0x15554c9ef900 <ompi_mpi_2dblprec>,
op=0x15554ca28980 <ompi_mpi_op_maxloc>, comm=0x7f1e30, module=0xaecb80)
at coll_hcoll_ops.c:217
#5 0x000015554c59811a in PMPI_Allreduce (sendbuf=0x7fffffff9e20,
recvbuf=0x7fffffff9e30, count=1, datatype=0x15554c9ef900 <ompi_mpi_2dblprec>, op=0x15554ca28980 <ompi_mpi_op_maxloc>, comm=0x7f1e30) at allreduce.c:123
#6 0x0000155553eabede in MPIU_Allreduce_Private () from /home/cklaij/ReFRESCO/trunk/install/extLibs/lib/libpetsc.so.3.22
#7 0x0000155553e50d08 in PetscPrintXMLNestedLinePerfResults () from /home/cklaij/ReFRESCO/trunk/install/extLibs/lib/libpetsc.so.3.22
#8 0x0000155553e5123e in PetscLogNestedTreePrintLine () from /home/cklaij/ReFRESCO/trunk/install/extLibs/lib/libpetsc.so.3.22
#9 0x0000155553e51f3a in PetscLogNestedTreePrint () from /home/cklaij/ReFRESCO/trunk/install/extLibs/lib/libpetsc.so.3.22
#10 0x0000155553e51e96 in PetscLogNestedTreePrint () from /home/cklaij/ReFRESCO/trunk/install/extLibs/lib/libpetsc.so.3.22
#11 0x0000155553e51e96 in PetscLogNestedTreePrint () from /home/cklaij/ReFRESCO/trunk/install/extLibs/lib/libpetsc.so.3.22
#12 0x0000155553e52142 in PetscLogNestedTreePrintTop () from /home/cklaij/ReFRESCO/trunk/install/extLibs/lib/libpetsc.so.3.22
#13 0x0000155553e5257b in PetscLogHandlerView_Nested_XML () from /home/cklaij/ReFRESCO/trunk/install/extLibs/lib/libpetsc.so.3.22
#14 0x0000155553e4e5a0 in PetscLogHandlerView_Nested () from /home/cklaij/ReFRESCO/trunk/install/extLibs/lib/libpetsc.so.3.22
#15 0x0000155553e56232 in PetscLogHandlerView () from /home/cklaij/ReFRESCO/trunk/install/extLibs/lib/libpetsc.so.3.22
#16 0x0000155553e588c3 in PetscLogView () from /home/cklaij/ReFRESCO/trunk/install/extLibs/lib/libpetsc.so.3.22
#17 0x0000155553e40eb5 in petsclogview_ () from /home/cklaij/ReFRESCO/trunk/install/extLibs/lib/libpetsc.so.3.22
#18 0x0000000000402c8b in MAIN__ ()
#19 0x00000000004023df in main ()
[cid:ii_197ebccaa1d27ee6ef21]
dr. ir.         Christiaan       Klaij   |      senior researcher
Research & Development   |      CFD Development
T +31 317 49 33 44<tel:+31%20317%2049%2033%2044>         |      https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!bV-wSzFdop4YFdAYdaN5SqxF1Ea1onC_aTyxTwYg3TAYhYJpnIuO7MDbEv5VmkfbhJpB3EHweorcvDLsr_38Wj4$ <https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!dAFNrWR8FzE9RrQXQAlok1iR_fA-rZdm9JAi-dlnKTnbdNTOTCViw0Nc-jjU4g72I-mhE1x1MZaf8imk4ivm_tE$>
[Facebook]<https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!dAFNrWR8FzE9RrQXQAlok1iR_fA-rZdm9JAi-dlnKTnbdNTOTCViw0Nc-jjU4g72I-mhE1x1MZaf8imkLNCvsiI$>
[LinkedIn]<https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!dAFNrWR8FzE9RrQXQAlok1iR_fA-rZdm9JAi-dlnKTnbdNTOTCViw0Nc-jjU4g72I-mhE1x1MZaf8imkrb79Ay4$>
[YouTube]<https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!dAFNrWR8FzE9RrQXQAlok1iR_fA-rZdm9JAi-dlnKTnbdNTOTCViw0Nc-jjU4g72I-mhE1x1MZaf8imkJiCoeLw$>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250710/429fe052/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image433266.png
Type: image/png
Size: 5004 bytes
Desc: image433266.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250710/429fe052/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image106439.png
Type: image/png
Size: 487 bytes
Desc: image106439.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250710/429fe052/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image416923.png
Type: image/png
Size: 504 bytes
Desc: image416923.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250710/429fe052/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image856574.png
Type: image/png
Size: 482 bytes
Desc: image856574.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250710/429fe052/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image043083.png
Type: image/png
Size: 5004 bytes
Desc: image043083.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250710/429fe052/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image954569.png
Type: image/png
Size: 487 bytes
Desc: image954569.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250710/429fe052/attachment-0013.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image488066.png
Type: image/png
Size: 504 bytes
Desc: image488066.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250710/429fe052/attachment-0014.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image817580.png
Type: image/png
Size: 482 bytes
Desc: image817580.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250710/429fe052/attachment-0015.png>