<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> dazza simplythebest <sayosale@hotmail.com><br>
<b>Sent:</b> Wednesday, August 25, 2021 12:08 PM<br>
<b>To:</b> Matthew Knepley <knepley@gmail.com><br>
<b>Subject:</b> Re: [petsc-users] Improving efficiency of slepc usage</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:18pt">​</span>Dear Matthew and Jose,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I have derived a smaller program from the original program by constructing
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
matrices of the same size, but filling their entries randomly instead of computing the correct
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
fluid dynamics values just to allow faster experimentation. This modified code's behaviour seems
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
to be similar, with the code again failing for the large matrix case with the SIGKILL error, so I first report
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
results from that code here. Firstly I can confirm that I am using Fortran , and I am compiling with the
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
intel compiler, which it seems places automatic arrays on the stack. The stacksize, as determined
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
by ulimit -a, is reported to be :<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
stack size (kbytes, -s) 8192<br>
</div>
<div>
<div id="x_appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[1] Okay, so I followed your suggestion and used ctrl-c followed by 'where' in one of the non-SIGKILL gdb windows.
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I have pasted the output into the bottom of this email (see [1] output) - it does look like the problem occurs somewhere in the call</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
to the MUMPS solver ?</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[2] I have also today gained access to another workstation, and so have tried running the (original) code on that machine.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
This new machine has two (more powerful) CPU nodes and a larger memory (both machines feature Intel Xeon processors).</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
On this new machine the large matrix case again failed with the familiar SIGKILL report when I used 16 or 12 MPI</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
processes, ran to the end w/out error for 4 or 6 MPI processes, <b>and failed but with a PETSC error message
</b><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
when I used 8 MPI processes, which I have pasted below (see [2] output). Does this point to some sort of resource</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
demand that exceeds some limit as the number of MPI processes increases ?</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Many thanks once again,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Dan.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:14pt">[</span><b><span style="font-size:14pt">2] output</span></b><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(0,36,81)">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: Error in external library</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-9, INFO(2)=6</span></div>
<div><br>
</div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021
</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: ./stab1.exe on a arch-omp_nodbug named super02 by darren Wed Aug 25 11:18:48 2021</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: Configure options ----with-debugging=0--package-prefix-hash=/home/darren/petsc-hash-pkgs --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-mpiexec=mpiexec.hydra COPTFLAGS="-g -O" FOPTFLAGS="-g
-O" CXXOPTFLAGS="-g -O" --with-64-bit-indices=1 --with-scalar-type=complex --with-precision=double --with-debugging=0 --with-openmp --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --with-mkl_pardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl
--with-mkl_cpardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --download-mumps --download-scalapack --download-cmake PETSC_ARCH=arch-omp_nodbug</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: #1 MatFactorNumeric_MUMPS() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/impls/aij/mpi/mumps/mumps.c:1686</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: Error in external library</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-9, INFO(2)=6</span></div>
<div><br>
</div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021
</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: ./stab1.exe on a arch-omp_nodbug named super02 by darren Wed Aug 25 11:18:48 2021</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: Configure options ----with-debugging=0--package-prefix-hash=/home/darren/petsc-hash-pkgs --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-mpiexec=mpiexec.hydra COPTFLAGS="-g -O" FOPTFLAGS="-g
-O" CXXOPTFLAGS="-g -O" --with-64-bit-indices=1 --with-scalar-type=complex --with-precision=double --with-debugging=0 --with-openmp --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --with-mkl_pardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl
--with-mkl_cpardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --download-mumps --download-scalapack --download-cmake PETSC_ARCH=arch-omp_nodbug</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: #1 MatFactorNumeric_MUMPS() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/impls/aij/mpi/mumps/mumps.c:1686</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: #2 MatLUFactorNumeric() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/interface/matrix.c:3195</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: #3 PCSetUp_LU() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/impls/factor/lu/lu.c:131</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: #4 PCSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/interface/precon.c:1015</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: #5 KSPSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/ksp/interface/itfunc.c:406</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: Error in external library</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-9, INFO(2)=6</span></div>
<div><br>
</div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021
</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: ./stab1.exe on a arch-omp_nodbug named super02 by darren Wed Aug 25 11:18:48 2021</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: Configure options ----with-debugging=0--package-prefix-hash=/home/darren/petsc-hash-pkgs --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-mpiexec=mpiexec.hydra COPTFLAGS="-g -O" FOPTFLAGS="-g
-O" CXXOPTFLAGS="-g -O" --with-64-bit-indices=1 --with-scalar-type=complex --with-precision=double --with-debugging=0 --with-openmp --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --with-mkl_pardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl
--with-mkl_cpardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --download-mumps --download-scalapack --download-cmake PETSC_ARCH=arch-omp_nodbug</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: #1 MatFactorNumeric_MUMPS() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/impls/aij/mpi/mumps/mumps.c:1686</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: #2 MatLUFactorNumeric() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/interface/matrix.c:3195</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: #3 PCSetUp_LU() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/impls/factor/lu/lu.c:131</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: #4 PCSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/interface/precon.c:1015</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: #5 KSPSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/ksp/interface/itfunc.c:406</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: Error in external library</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-9, INFO(2)=6</span></div>
<div><br>
</div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021
</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: ./stab1.exe on a arch-omp_nodbug named super02 by darren Wed Aug 25 11:18:48 2021</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: Configure options ----with-debugging=0--package-prefix-hash=/home/darren/petsc-hash-pkgs --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-mpiexec=mpiexec.hydra COPTFLAGS="-g -O" FOPTFLAGS="-g
-O" CXXOPTFLAGS="-g -O" --with-64-bit-indices=1 --with-scalar-type=complex --with-precision=double --with-debugging=0 --with-openmp --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --with-mkl_pardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl
--with-mkl_cpardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --download-mumps --download-scalapack --download-cmake PETSC_ARCH=arch-omp_nodbug</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: #1 MatFactorNumeric_MUMPS() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/impls/aij/mpi/mumps/mumps.c:1686</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: #2 MatLUFactorNumeric() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/interface/matrix.c:3195</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: #3 PCSetUp_LU() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/impls/factor/lu/lu.c:131</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: #4 PCSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/interface/precon.c:1015</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: #5 KSPSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/ksp/interface/itfunc.c:406</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: #6 STSetUp_Sinvert() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/impls/sinvert/sinvert.c:123</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: #7 STSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/interface/stsolve.c:582</span></div>
<div><span style="color:rgb(0,36,81)">[3]PETSC ERROR: #8 EPSSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssetup.c:350</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: Error in external library</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-9, INFO(2)=6</span></div>
<div><br>
</div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021
</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: ./stab1.exe on a arch-omp_nodbug named super02 by darren Wed Aug 25 11:18:48 2021</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: Configure options ----with-debugging=0--package-prefix-hash=/home/darren/petsc-hash-pkgs --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-mpiexec=mpiexec.hydra COPTFLAGS="-g -O" FOPTFLAGS="-g
-O" CXXOPTFLAGS="-g -O" --with-64-bit-indices=1 --with-scalar-type=complex --with-precision=double --with-debugging=0 --with-openmp --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --with-mkl_pardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl
--with-mkl_cpardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --download-mumps --download-scalapack --download-cmake PETSC_ARCH=arch-omp_nodbug</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: #1 MatFactorNumeric_MUMPS() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/impls/aij/mpi/mumps/mumps.c:1686</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: #2 MatLUFactorNumeric() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/interface/matrix.c:3195</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: #3 PCSetUp_LU() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/impls/factor/lu/lu.c:131</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: #4 PCSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/interface/precon.c:1015</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: #5 KSPSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/ksp/interface/itfunc.c:406</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: #6 STSetUp_Sinvert() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/impls/sinvert/sinvert.c:123</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: #7 STSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/interface/stsolve.c:582</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: #8 EPSSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssetup.c:350</span></div>
<div><span style="color:rgb(0,36,81)">[4]PETSC ERROR: #9 EPSSolve() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssolve.c:136</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: Error in external library</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-9, INFO(2)=6</span></div>
<div><br>
</div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021
</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: ./stab1.exe on a arch-omp_nodbug named super02 by darren Wed Aug 25 11:18:48 2021</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: Configure options ----with-debugging=0--package-prefix-hash=/home/darren/petsc-hash-pkgs --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-mpiexec=mpiexec.hydra COPTFLAGS="-g -O" FOPTFLAGS="-g
-O" CXXOPTFLAGS="-g -O" --with-64-bit-indices=1 --with-scalar-type=complex --with-precision=double --with-debugging=0 --with-openmp --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --with-mkl_pardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl
--with-mkl_cpardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --download-mumps --download-scalapack --download-cmake PETSC_ARCH=arch-omp_nodbug</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: #1 MatFactorNumeric_MUMPS() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/impls/aij/mpi/mumps/mumps.c:1686</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: #2 MatLUFactorNumeric() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/interface/matrix.c:3195</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: #3 PCSetUp_LU() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/impls/factor/lu/lu.c:131</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: #4 PCSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/interface/precon.c:1015</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: #5 KSPSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/ksp/interface/itfunc.c:406</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: #6 STSetUp_Sinvert() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/impls/sinvert/sinvert.c:123</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: #7 STSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/interface/stsolve.c:582</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: #8 EPSSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssetup.c:350</span></div>
<div><span style="color:rgb(0,36,81)">[5]PETSC ERROR: #9 EPSSolve() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssolve.c:136</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: Error in external library</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-9, INFO(2)=21891045</span></div>
<div><br>
</div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021
</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: ./stab1.exe on a arch-omp_nodbug named super02 by darren Wed Aug 25 11:18:48 2021</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: Configure options ----with-debugging=0--package-prefix-hash=/home/darren/petsc-hash-pkgs --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-mpiexec=mpiexec.hydra COPTFLAGS="-g -O" FOPTFLAGS="-g
-O" CXXOPTFLAGS="-g -O" --with-64-bit-indices=1 --with-scalar-type=complex --with-precision=double --with-debugging=0 --with-openmp --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --with-mkl_pardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl
--with-mkl_cpardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --download-mumps --download-scalapack --download-cmake PETSC_ARCH=arch-omp_nodbug</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/impls/aij/mpi/mumps/mumps.c:1686</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: #2 MatLUFactorNumeric() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/interface/matrix.c:3195</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: #3 PCSetUp_LU() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/impls/factor/lu/lu.c:131</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: #4 PCSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/interface/precon.c:1015</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: #5 KSPSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/ksp/interface/itfunc.c:406</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: #6 STSetUp_Sinvert() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/impls/sinvert/sinvert.c:123</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: #7 STSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/interface/stsolve.c:582</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: #8 EPSSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssetup.c:350</span></div>
<div><span style="color:rgb(0,36,81)">[6]PETSC ERROR: #9 EPSSolve() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssolve.c:136</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: Error in external library</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-9, INFO(2)=21841925</span></div>
<div><br>
</div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021
</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: ./stab1.exe on a arch-omp_nodbug named super02 by darren Wed Aug 25 11:18:48 2021</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: Configure options ----with-debugging=0--package-prefix-hash=/home/darren/petsc-hash-pkgs --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-mpiexec=mpiexec.hydra COPTFLAGS="-g -O" FOPTFLAGS="-g
-O" CXXOPTFLAGS="-g -O" --with-64-bit-indices=1 --with-scalar-type=complex --with-precision=double --with-debugging=0 --with-openmp --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --with-mkl_pardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl
--with-mkl_cpardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --download-mumps --download-scalapack --download-cmake PETSC_ARCH=arch-omp_nodbug</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/impls/aij/mpi/mumps/mumps.c:1686</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: #2 MatLUFactorNumeric() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/interface/matrix.c:3195</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: #3 PCSetUp_LU() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/impls/factor/lu/lu.c:131</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: #4 PCSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/interface/precon.c:1015</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: #5 KSPSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/ksp/interface/itfunc.c:406</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: #6 STSetUp_Sinvert() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/impls/sinvert/sinvert.c:123</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: #7 STSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/interface/stsolve.c:582</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: #8 EPSSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssetup.c:350</span></div>
<div><span style="color:rgb(0,36,81)">[7]PETSC ERROR: #9 EPSSolve() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssolve.c:136</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: #2 MatLUFactorNumeric() at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/interface/matrix.c:3195</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: #3 PCSetUp_LU() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/impls/factor/lu/lu.c:131</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: #4 PCSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/interface/precon.c:1015</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: #5 KSPSetUp() at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/ksp/interface/itfunc.c:406</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: #6 STSetUp_Sinvert() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/impls/sinvert/sinvert.c:123</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: #7 STSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/interface/stsolve.c:582</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: #8 EPSSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssetup.c:350</span></div>
<div><span style="color:rgb(0,36,81)">[0]PETSC ERROR: #9 EPSSolve() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssolve.c:136</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: #6 STSetUp_Sinvert() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/impls/sinvert/sinvert.c:123</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: #7 STSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/interface/stsolve.c:582</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: #8 EPSSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssetup.c:350</span></div>
<div><span style="color:rgb(0,36,81)">[1]PETSC ERROR: #9 EPSSolve() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssolve.c:136</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: #6 STSetUp_Sinvert() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/impls/sinvert/sinvert.c:123</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: #7 STSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/interface/stsolve.c:582</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: #8 EPSSetUp() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssetup.c:350</span></div>
<div><span style="color:rgb(0,36,81)">[2]PETSC ERROR: #9 EPSSolve() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssolve.c:136</span></div>
<span style="color:rgb(0,36,81)">[3]PETSC ERROR: #9 EPSSolve() at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssolve.c:136</span><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:14pt">[</span><b><span style="font-size:14pt">1] output</span></b><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(0,36,81)">Continuing.</span>
<div><span style="color:rgb(0,36,81)">[New Thread 0x7f6f5b2d2780 (LWP 794037)]</span></div>
<div><span style="color:rgb(0,36,81)">[New Thread 0x7f6f5aad0800 (LWP 794040)]</span></div>
<div><span style="color:rgb(0,36,81)">[New Thread 0x7f6f5a2ce880 (LWP 794041)]</span></div>
<div><span style="color:rgb(0,36,81)">^C</span></div>
<div><span style="color:rgb(0,36,81)">Thread 1 "my.exe" received signal SIGINT, Interrupt.</span></div>
<div><span style="color:rgb(0,36,81)">0x00007f72904927b0 in ofi_fastlock_release_noop ()</span></div>
<div><span style="color:rgb(0,36,81)"> from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/libtcp-fi.so</span></div>
<div><span style="color:rgb(0,36,81)">(gdb) where</span></div>
<div><span style="color:rgb(0,36,81)">#0 0x00007f72904927b0 in ofi_fastlock_release_noop ()</span></div>
<div><span style="color:rgb(0,36,81)"> from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/libtcp-fi.so</span></div>
<div><span style="color:rgb(0,36,81)">#1 0x00007f729049354b in ofi_cq_readfrom ()</span></div>
<div><span style="color:rgb(0,36,81)"> from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/libtcp-fi.so</span></div>
<div><span style="color:rgb(0,36,81)">#2 0x00007f728ffe8f0e in rxm_ep_do_progress ()</span></div>
<div><span style="color:rgb(0,36,81)"> from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/librxm-fi.so</span></div>
<div><span style="color:rgb(0,36,81)">#3 0x00007f728ffe2b7d in rxm_ep_recv_common_flags ()</span></div>
<div><span style="color:rgb(0,36,81)"> from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/librxm-fi.so</span></div>
<div><span style="color:rgb(0,36,81)">#4 0x00007f728ffe30f8 in rxm_ep_trecvmsg ()</span></div>
<div><span style="color:rgb(0,36,81)"> from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/librxm-fi.so</span></div>
<div><span style="color:rgb(0,36,81)">#5 0x00007f72fe6b8c3e in PMPI_Iprobe (source=14090824, tag=-1481647392,
</span></div>
<div><span style="color:rgb(0,36,81)"> comm=1, flag=0x0, status=0xffffffffffffffff)</span></div>
<div><span style="color:rgb(0,36,81)"> at /usr/include/rdma/fi_tagged.h:109</span></div>
<div><span style="color:rgb(0,36,81)">#6 0x00007f72ff3d7fad in pmpi_iprobe_ (v1=0xd70248, v2=0x7ffda7afdae0,
</span></div>
<div><span style="color:rgb(0,36,81)"> v3=0x1, v4=0x0, v5=0xffffffffffffffff, ierr=0xd6fc90)</span></div>
<div><span style="color:rgb(0,36,81)"> at ../../src/binding/fortran/mpif_h/iprobef.c:276</span></div>
<div><span style="color:rgb(0,36,81)">#7 0x00007f730855b6e2 in zmumps_try_recvtreat (comm_load=1, ass_irecv=0,
</span></div>
<div><span style="color:rgb(0,36,81)"> blocking=<error reading variable: Cannot access memory at address 0x1>,
</span></div>
<div><span style="color:rgb(0,36,81)"> </span></div>
<div><span style="color:rgb(0,36,81)"> --Type <RET> for more, q to quit, c to continue without paging--cont</span></div>
<div><span style="color:rgb(0,36,81)"> irecv=<error reading variable: Cannot access memory at address 0x0>, message_received=<error reading variable: Cannot access memory at address 0xffffffffffffffff>, msgsou=1, msgtag=-1, status=..., bufr=..., lbufr=320782504,
lbufr_bytes=1283130016, procnode_steps=..., posfac=1, iwpos=1, iwposcb=292535, iptrlu=2039063816, lrlu=2039063816, lrlus=2039063816, n=50400, iw=..., liw=292563, a=..., la=2611636796, ptrist=..., ptlust=..., ptrfac=..., ptrast=..., step=..., pimaster=...,
pamaster=..., nstk_s=..., comp=0, iflag=0, ierror=0, comm=-1006632958, nbprocfils=..., ipool=..., lpool=5, leaf=1, nbfin=4, myid=1, slavef=4, root=<error reading variable: value of type `zmumps_root_struc' requires 766016 bytes, which is more than max-value-size>,
opassw=0, opeliw=0, itloc=..., rhs_mumps=..., fils=..., dad=..., ptrarw=..., ptraiw=..., intarr=..., dblarr=..., icntl=..., keep=..., keep8=..., dkeep=..., nd=..., frere=..., lptrar=50400, nelt=1, frtptr=..., frtelt=..., istep_to_iniv2=..., tab_pos_in_pere=...,
stack_right_authorized=4294967295, lrgroups=...) at zfac_process_message.F:730</span></div>
<div><span style="color:rgb(0,36,81)">#8 0x00007f73087076e2 in zmumps_fac_par_m::zmumps_fac_par (n=1, iw=..., liw=<error reading variable: Cannot access memory at address 0x1>, a=..., la=<error reading variable: Cannot access memory at address 0xffffffffffffffff>,
nstk_steps=..., nbprocfils=..., nd=..., fils=..., step=..., frere=..., dad=..., cand=..., istep_to_iniv2=..., tab_pos_in_pere=..., nstepsdone=1690339657, opass=<error reading variable: Cannot access memory at address 0x5>, opeli=<error reading variable: Cannot
access memory at address 0x0>, nelva=50400, comp=259581, maxfrt=-1889517576, nmaxnpiv=-1195144887, ntotpv=<error reading variable: Cannot access memory at address 0x2>, noffnegpv=<error reading variable: Cannot access memory at address 0x0>, nb22t1=<error
reading variable: Cannot access memory at address 0x0>, nb22t2=<error reading variable: Cannot access memory at address 0x0>, nbtiny=<error reading variable: Cannot access memory at address 0x0>, det_exp=<error reading variable: Cannot access memory at address
0x0>, det_mant=<error reading variable: Cannot access memory at address 0x0>, det_sign=<error reading variable: Cannot access memory at address 0x0>, ptrist=..., ptrast=..., pimaster=..., pamaster=..., ptrarw=..., ptraiw=..., itloc=..., rhs_mumps=..., ipool=...,
lpool=<error reading variable: Cannot access memory at address 0x0>, rinfo=<error reading variable: Cannot access memory at address 0x0>, posfac=<error reading variable: Cannot access memory at address 0x0>, iwpos=<error reading variable: Cannot access memory
at address 0x0>, lrlu=<error reading variable: Cannot access memory at address 0x0>, iptrlu=<error reading variable: Cannot access memory at address 0x0>, lrlus=<error reading variable: Cannot access memory at address 0x0>, leaf=<error reading variable: Cannot
access memory at address 0x0>, nbroot=<error reading variable: Cannot access memory at address 0x0>, nbrtot=<error reading variable: Cannot access memory at address 0x0>, uu=<error reading variable: Cannot access memory at address 0x0>, icntl=<error reading
variable: Cannot access memory at address 0x0>, ptlust=..., ptrfac=..., info=<error reading variable: Cannot access memory at address 0x0>, keep=<error reading variable: Cannot access memory at address 0x3ff0000000000000>, keep8=<error reading variable: Cannot
access memory at address 0x0>, procnode_steps=..., slavef=<error reading variable: Cannot access memory at address 0x4ffffffff>, myid=<error reading variable: Cannot access memory at address 0xffffffff>, comm_nodes=<error reading variable: Cannot access memory
at address 0x0>, myid_nodes=<error reading variable: Cannot access memory at address 0x0>, bufr=..., lbufr=0, lbufr_bytes=5, intarr=..., dblarr=..., root=..., perm=..., nelt=0, frtptr=..., frtelt=..., lptrar=3, comm_load=-30, ass_irecv=30, seuil=2.1219957909652723e-314,
seuil_ldlt_niv2=4.2439866417681519e-314, mem_distrib=..., ne=..., dkeep=..., pivnul_list=..., lpn_list=0, lrgroups=...) at zfac_par_m.F:182</span></div>
<div><span style="color:rgb(0,36,81)">#9 0x00007f730865af7a in zmumps_fac_b (n=1, s_is_pointers=..., la=<error reading variable: Cannot access memory at address 0x1>, liw=<error reading variable: Cannot access memory at address 0x0>, sym_perm=..., na=...,
lna=1, ne_steps=..., nfsiz=..., fils=..., step=..., frere=..., dad=..., cand=..., istep_to_iniv2=..., tab_pos_in_pere=..., ptrar=..., ldptrar=<error reading variable: Cannot access memory at address 0x0>, ptrist=..., ptlust_s=..., ptrfac=..., iw1=..., iw2=...,
itloc=..., rhs_mumps=..., pool=..., lpool=-1889529280, cntl1=-5.3576889161551131e-255, icntl=<error reading variable: Cannot access memory at address 0x25344>, info=..., rinfo=..., keep=..., keep8=..., procnode_steps=..., slavef=-1889504640, comm_nodes=-2048052411,
myid=<error reading variable: Cannot access memory at address 0x81160>, myid_nodes=-1683330500, bufr=..., lbufr=<error reading variable: Cannot access memory at address 0x11db4c>, lbufr_bytes=<error reading variable: Cannot access memory at address 0xc4e0>,
zmumps_lbuf=<error reading variable: Cannot access memory at address 0x4>, intarr=..., dblarr=..., root=<error reading variable: Cannot access memory at address 0x11dbec>, nelt=<error reading variable: Cannot access memory at address 0x3>, frtptr=..., frtelt=...,
comm_load=<error reading variable: Cannot access memory at address 0x0>, ass_irecv=<error reading variable: Cannot access memory at address 0x0>, seuil=<error reading variable: Cannot access memory at address 0x0>, seuil_ldlt_niv2=<error reading variable:
Cannot access memory at address 0x0>, mem_distrib=<error reading variable: Cannot access memory at address 0x0>, dkeep=<error reading variable: Cannot access memory at address 0x0>, pivnul_list=..., lpn_list=<error reading variable: Cannot access memory at
address 0x0>, lrgroups=...) at zfac_b.F:243</span></div>
<div><span style="color:rgb(0,36,81)">#10 0x00007f7308610ff7 in zmumps_fac_driver (id=<error reading variable: value of type `zmumps_struc' requires 386095520 bytes, which is more than max-value-size>) at zfac_driver.F:2421</span></div>
<div><span style="color:rgb(0,36,81)">#11 0x00007f7308569256 in zmumps (id=<error reading variable: value of type `zmumps_struc' requires 386095520 bytes, which is more than max-value-size>) at zmumps_driver.F:1883</span></div>
<div><span style="color:rgb(0,36,81)">#12 0x00007f73084cf756 in zmumps_f77 (job=1, sym=0, par=<error reading variable: Cannot access memory at address 0x1>, comm_f77=<error reading variable: Cannot access memory at address 0x0>, n=<error reading variable: Cannot
access memory at address 0xffffffffffffffff>, nblk=1, icntl=..., cntl=..., keep=..., dkeep=..., keep8=..., nz=0, nnz=0, irn=..., irnhere=0, jcn=..., jcnhere=0, a=..., ahere=0, nz_loc=0, nnz_loc=304384739, irn_loc=..., irn_lochere=1, jcn_loc=..., jcn_lochere=1,
a_loc=..., a_lochere=1, nelt=0, eltptr=..., eltptrhere=0, eltvar=..., eltvarhere=0, a_elt=..., a_elthere=0, blkptr=..., blkptrhere=0, blkvar=..., blkvarhere=0, perm_in=..., perm_inhere=0, rhs=..., rhshere=0, redrhs=..., redrhshere=0, info=..., rinfo=..., infog=...,
rinfog=..., deficiency=0, lwk_user=0, size_schur=0, listvar_schur=..., listvar_schurhere=0, schur=..., schurhere=0, wk_user=..., wk_userhere=0, colsca=..., colscahere=0, rowsca=..., rowscahere=0, instance_number=1, nrhs=1, lrhs=0, lredrhs=0, rhs_sparse=...,
rhs_sparsehere=0, sol_loc=..., sol_lochere=0, rhs_loc=..., rhs_lochere=0, irhs_sparse=..., irhs_sparsehere=0, irhs_ptr=..., irhs_ptrhere=0, isol_loc=..., isol_lochere=0, irhs_loc=..., irhs_lochere=0, nz_rhs=0, lsol_loc=0, lrhs_loc=0, nloc_rhs=0, schur_mloc=0,
schur_nloc=0, schur_lld=0, mblock=0, nblock=0, nprow=0, npcol=0, ooc_tmpdir=..., ooc_prefix=..., write_problem=..., save_dir=..., save_prefix=..., tmpdirlen=20, prefixlen=20, write_problemlen=20, save_dirlen=20, save_prefixlen=20, metis_options=...) at zmumps_f77.F:289</span></div>
<div><span style="color:rgb(0,36,81)">#13 0x00007f73084cd391 in zmumps_c (mumps_par=0xd70248) at mumps_c.c:485</span></div>
<div><span style="color:rgb(0,36,81)">#14 0x00007f7307c035ad in MatFactorNumeric_MUMPS (F=0xd70248, A=0x7ffda7afdae0, info=0x1) at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/impls/aij/mpi/mumps/mumps.c:1683</span></div>
<div><span style="color:rgb(0,36,81)">#15 0x00007f7307765a8b in MatLUFactorNumeric (fact=0xd70248, mat=0x7ffda7afdae0, info=0x1) at /data/work/slepc/PETSC/petsc-3.15.0/src/mat/interface/matrix.c:3195</span></div>
<div><span style="color:rgb(0,36,81)">#16 0x00007f73081b8427 in PCSetUp_LU (pc=0xd70248) at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/impls/factor/lu/lu.c:131</span></div>
<div><span style="color:rgb(0,36,81)">#17 0x00007f7308214939 in PCSetUp (pc=0xd70248) at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/pc/interface/precon.c:1015</span></div>
<div><span style="color:rgb(0,36,81)">#18 0x00007f73082260ae in KSPSetUp (ksp=0xd70248) at /data/work/slepc/PETSC/petsc-3.15.0/src/ksp/ksp/interface/itfunc.c:406</span></div>
<div><span style="color:rgb(0,36,81)">#19 0x00007f7309114959 in STSetUp_Sinvert (st=0xd70248) at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/impls/sinvert/sinvert.c:123</span></div>
<div><span style="color:rgb(0,36,81)">#20 0x00007f7309130462 in STSetUp (st=0xd70248) at /data/work/slepc/SLEPC/slepc-3.15.1/src/sys/classes/st/interface/stsolve.c:582</span></div>
<div><span style="color:rgb(0,36,81)">#21 0x00007f73092504af in EPSSetUp (eps=0xd70248) at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssetup.c:350</span></div>
<div><span style="color:rgb(0,36,81)">#22 0x00007f7309253635 in EPSSolve (eps=0xd70248) at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/epssolve.c:136</span></div>
<div><span style="color:rgb(0,36,81)">#23 0x00007f7309259c8d in epssolve_ (eps=0xd70248, __ierr=0x7ffda7afdae0) at /data/work/slepc/SLEPC/slepc-3.15.1/src/eps/interface/ftn-auto/epssolvef.c:85</span></div>
<div><span style="color:rgb(0,36,81)">#24 0x0000000000403c19 in all_stab_routines::solve_by_slepc2 (a_pet=..., b_pet=..., jthisone=<error reading variable: Cannot access memory at address 0x1>, isize=<error reading variable: Cannot access memory at address
0x0>) at small_slepc_example_program.F:322</span></div>
<div><span style="color:rgb(0,36,81)">#25 0x00000000004025a0 in slepit () at small_slepc_example_program.F:549</span></div>
<div><span style="color:rgb(0,36,81)">#26 0x00000000004023f2 in main ()</span></div>
<div><span style="color:rgb(0,36,81)">#27 0x00007f72fb8380b3 in __libc_start_main (main=0x4023c0 <main>, argc=14, argv=0x7ffda7b024e8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffda7b024d8) at ../csu/libc-start.c:308</span></div>
<span style="color:rgb(0,36,81)">#28 0x00000000004022fe in _start ()</span><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Matthew Knepley <knepley@gmail.com><br>
<b>Sent:</b> Tuesday, August 24, 2021 3:59 PM<br>
<b>To:</b> dazza simplythebest <sayosale@hotmail.com><br>
<b>Cc:</b> Jose E. Roman <jroman@dsic.upv.es>; PETSc <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Improving efficiency of slepc usage</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Tue, Aug 24, 2021 at 8:47 AM dazza simplythebest <<a href="mailto:sayosale@hotmail.com">sayosale@hotmail.com</a>> wrote:<br>
</div>
<div class="x_x_gmail_quote">
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div>
<div id="x_x_gmail-m_3253114932700099937appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Dear Matthew and Jose,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Apologies for the delayed reply, I had a couple of unforeseen days off this week.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Firstly regarding Jose's suggestion re: MUMPS, the program is already using MUMPS</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
to solve linear systems (the code is using a distributed MPI matrix to solve the generalised
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
non-Hermitian complex problem).</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I have tried the gdb debugger as per Matthew's suggestion.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Just to note in case someone else is following this that at first it didn't work (couldn't 'attach') ,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
but after some googling I found a tip suggesting the command;</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span><code>echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope</code></span><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
which seemed to get it working.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<b>I then first ran the debugger on the small matrix case that worked.</b></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
That stopped in gdb almost immediately after starting execution <br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
with a report regarding 'nanosleep.c':<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(12,100,192)">../sysdeps/unix/sysv/linux/clock_nanosleep.c: No such file or directory.</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
However, issuing the 'cont' command again caused the program to run through to the end of the</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
execution w/out any problems, and with correct looking results, so I am guessing this error</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
is not particularly important.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>We do that on purpose when the debugger starts up. Typing 'cont' is correct.</div>
<div> </div>
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<b>I then tried the same debugging procedure on the large matrix case that fails.</b></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
The code again stopped almost immediately after the start of execution with <br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
the same nanosleep error as before, and I was able to set the program running <br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
again with 'cont' (see full output below). I was running the code with 4 MPI processes,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
and so had 4 gdb windows appear. Thereafter the code ran for sometime until completing the
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
matrix construction, and then one of the gdb process windows printed a <br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(12,100,192)">Program terminated with signal SIGKILL, Killed.</span><br>
<span style="color:rgb(12,100,192)">The program no longer exists.</span><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
message. I then typed 'where' into this terminal but just received the message</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(12,100,192)">No stack.</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I have only seen this behavior one other time, and it was with Fortran. Fortran allows you to declare really big arrays</div>
<div>on the stack by putting them at the start of a function (rather than F90 malloc). When I had one of those arrays exceed</div>
<div>the stack space, I got this kind of an error where everything is destroyed rather than just stopping. Could it be that you</div>
<div>have a large structure on the stack?</div>
<div><br>
</div>
<div>Second, you can at least look at the stack for the processes that were not killed. You type Ctrl-C, which should give you</div>
<div>the prompt and then "where".</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
The other gdb windows basically seemed to be left in limbo until I issued the 'quit'</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
command in the SIGKILL, and then they vanished.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I paste the full output from the gdb window that recorded the SIGKILL below here.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I guess it is necessary to somehow work out where the SIGKILL originates from ?</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks once again,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Dan.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(23,78,134)">GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2</span>
<div><span style="color:rgb(23,78,134)">Copyright (C) 2020 Free Software Foundation, Inc.</span></div>
<div><span style="color:rgb(23,78,134)">License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.html</a>></span></div>
<div><span style="color:rgb(23,78,134)">This is free software: you are free to change and redistribute it.</span></div>
<div><span style="color:rgb(23,78,134)">There is NO WARRANTY, to the extent permitted by law.</span></div>
<div><span style="color:rgb(23,78,134)">Type "show copying" and "show warranty" for details.</span></div>
<div><span style="color:rgb(23,78,134)">This GDB was configured as "x86_64-linux-gnu".</span></div>
<div><span style="color:rgb(23,78,134)">Type "show configuration" for configuration details.</span></div>
<div><span style="color:rgb(23,78,134)">For bug reporting instructions, please see:</span></div>
<div><span style="color:rgb(23,78,134)"><<a href="http://www.gnu.org/software/gdb/bugs/" target="_blank">http://www.gnu.org/software/gdb/bugs/</a>>.</span></div>
<div><span style="color:rgb(23,78,134)">Find the GDB manual and other documentation resources online at:</span></div>
<div><span style="color:rgb(23,78,134)"> <<a href="http://www.gnu.org/software/gdb/documentation/" target="_blank">http://www.gnu.org/software/gdb/documentation/</a>>.</span></div>
<div><br>
</div>
<div><span style="color:rgb(23,78,134)">For help, type "help".</span></div>
<div><span style="color:rgb(23,78,134)">Type "apropos word" to search for commands related to "word"...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from ./stab1.exe...</span></div>
<div><span style="color:rgb(23,78,134)">Attaching to program: /data/work/rotplane/omega_to_zero/stability/test/tmp10/tmp6/stab1.exe, process 675919</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /data/work/slepc/SLEPC/slepc-3.15.1/arch-omp_nodbug/lib/libslepc.so.3.15...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /data/work/slepc/PETSC/petsc-3.15.0/arch-omp_nodbug/lib/libpetsc.so.3.15...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mkl/lib--Type <RET> for more, q to quit, c to continue without paging--cont</span></div>
<div><span style="color:rgb(23,78,134)">/intel64_lin/libmkl_intel_lp64.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mkl/lib/intel64_lin/libmkl_core.so...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mkl/lib/intel64_lin/libmkl_intel_thread.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/mkl/lib/intel64_lin/libmkl_intel_thread.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mkl/lib/intel64_lin/libmkl_blacs_intelmpi_lp64.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/mkl/lib/intel64_lin/libmkl_blacs_intelmpi_lp64.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libiomp5.so...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libiomp5.dbg...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.31.so...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /usr/lib/debug/.build-id/e5/4761f7b554d0fcc1562959665d93dffbebdaf0.debug...</span></div>
<div><span style="color:rgb(23,78,134)">[Thread debugging using libthread_db enabled]</span></div>
<div><span style="color:rgb(23,78,134)">Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/libmpifort.so.12...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.so.12...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/release/libmpi.dbg...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/librt-2.31.so...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libifport.so.5...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libifport.so.5)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libimf.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libimf.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libsvml.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libsvml.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.31.so...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libirc.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libirc.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /lib/x86_64-linux-gnu/libgcc_s.so.1)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /usr/lib/x86_64-linux-gnu/libquadmath.so.0...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /usr/lib/x86_64-linux-gnu/libquadmath.so.0)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/libmpi_ilp64.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/lib/libmpi_ilp64.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.31.so...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libirng.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libirng.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libintlc.so.5...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/libintlc.so.5)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /lib64/ld-linux-x86-64.so.2...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.31.so...</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/libfabric.so.1...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/libfabric.so.1)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /usr/lib/x86_64-linux-gnu/libnuma.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /usr/lib/x86_64-linux-gnu/libnuma.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/libtcp-fi.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/libtcp-fi.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/libsockets-fi.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/libsockets-fi.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/librxm-fi.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/librxm-fi.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/libpsmx2-fi.so...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /opt/intel/compilers_and_libraries_2020.0.166/linux/mpi/intel64/libfabric/lib/prov/libpsmx2-fi.so)</span></div>
<div><span style="color:rgb(23,78,134)">Reading symbols from /usr/lib/x86_64-linux-gnu/libpsm2.so.2...</span></div>
<div><span style="color:rgb(23,78,134)">(No debugging symbols found in /usr/lib/x86_64-linux-gnu/libpsm2.so.2)</span></div>
<div><span style="color:rgb(23,78,134)">0x00007fac4d0d8334 in __GI___clock_nanosleep (clock_id=<optimized out>, clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffdc641a9a0, rem=rem@entry=0x7ffdc641a9a0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78</span></div>
<div><span style="color:rgb(23,78,134)">78 ../sysdeps/unix/sysv/linux/clock_nanosleep.c: No such file or directory.</span></div>
<div><span style="color:rgb(23,78,134)">(gdb) cont</span></div>
<div><span style="color:rgb(23,78,134)">Continuing.</span></div>
<div><span style="color:rgb(23,78,134)">[New Thread 0x7f9e49c02780 (LWP 676559)]</span></div>
<div><span style="color:rgb(23,78,134)">[New Thread 0x7f9e49400800 (LWP 676560)]</span></div>
<div><span style="color:rgb(23,78,134)">[New Thread 0x7f9e48bfe880 (LWP 676562)]</span></div>
<div><span style="color:rgb(23,78,134)">[Thread 0x7f9e48bfe880 (LWP 676562) exited]</span></div>
<div><span style="color:rgb(23,78,134)">[Thread 0x7f9e49400800 (LWP 676560) exited]</span></div>
<div><span style="color:rgb(23,78,134)">[Thread 0x7f9e49c02780 (LWP 676559) exited]</span></div>
<div><br>
</div>
<div><span style="color:rgb(23,78,134)">Program terminated with signal SIGKILL, Killed.</span></div>
<div><span style="color:rgb(23,78,134)">The program no longer exists.</span></div>
<div><span style="color:rgb(23,78,134)">(gdb) where</span></div>
<span style="color:rgb(23,78,134)">No stack.</span><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
<br>
</div>
<hr style="display:inline-block; width:98%">
<div id="x_x_gmail-m_3253114932700099937divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> Friday, August 20, 2021 2:12 PM<br>
<b>To:</b> dazza simplythebest <<a href="mailto:sayosale@hotmail.com" target="_blank">sayosale@hotmail.com</a>><br>
<b>Cc:</b> Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>>; PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Improving efficiency of slepc usage</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Fri, Aug 20, 2021 at 6:55 AM dazza simplythebest <<a href="mailto:sayosale@hotmail.com" target="_blank">sayosale@hotmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Dear Jose,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Many thanks for your response, I have been investigating this issue with a few more calculations
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
today, hence the slightly delayed response.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
The problem is actually derived from a fluid dynamics problem, so to allow an easier exploration of things
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I first downsized the resolution of the underlying fluid solver while keeping all the physical parameters</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
the same - i.e. I would get a smaller matrix that should be solving the same physical problem as the original</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
larger matrix but to lower accuracy. <br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<u><b>Results</b></u></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<b>Small matrix (N= 21168) - everything good!</b><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
This converged when using the -eps_largest_real approach (taking 92 iterations for nev=10,<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
tol= 5.0000E-06 and ncv = 300), and also when using the shift-invert approach, converging
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
very impressively in a single iteration ! Interestingly it did this both for a non-zero -eps_target<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
and also for a zero -eps_target.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div>
<div id="x_x_gmail-m_3253114932700099937x_gmail-m_-6879576846774134416appendonsend">
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<b>Large matrix (N=50400)- works for -eps_largest_real , fails for st_type sinvert
</b><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I have just double checked again that the code does run properly when we use the -eps_largest_real
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
option - indeed I ran it with a small nev and large tolerance (nev = 4, tol= -eps_tol 5.0e-4 , ncv = 300)</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
and with these parameters convergence was obtained in 164 iterations, which took 6 hours on the
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
machine I was running it on. Furthermore the eigenvalues seem to be ballpark correct; for this large</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
higher resolution case (although with lower slepc tolerance) we obtain 1789.56816314173 -4724.51319554773i</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
as the eigenvalue with largest real part, while the smaller matrix (same physical problem but at lower resolution case)</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
found this eigenvalue to be 1831.11845726501 -4787.54519511345i , which means the agreement is in line</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
with expectations.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<i>Unfortunately though the code does still crash though when I try to do shift-invert for the large matrix case
</i>,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
whether or not I use a non-zero -eps_target. For reference this is the command line used :</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
-eps_nev 10 -eps_ncv 300 -log_view -eps_view -eps_target 0.1 -st_type sinvert -eps_monitor :monitor_output05.txt
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
To be precise the code crashes soon after calling EPSSolve (it successfully calls
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
MatCreateVecs, EPSCreate, EPSSetOperators, EPSSetProblemType and EPSSetFromOptions).</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
By crashes I mean that I do not even get any error messages from slepc/PETSC, and do not even get the
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
'EPS Object: 16 MPI processes' message - I simply get a MPI/Fortran 'KILLED BY SIGNAL: 9 (Killed)' message</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
as soon as EPSsolve is called.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Hi Dan,</div>
<div><br>
</div>
<div>It would help track this error down if we had a stack trace. You can get a stack trace from the debugger. You run with</div>
<div><br>
</div>
<div> -start_in_debugger</div>
<div><br>
</div>
<div>which should launch the debugger (usually), and then type</div>
<div><br>
</div>
<div> cont</div>
<div><br>
</div>
<div>to continue, and then</div>
<div><br>
</div>
<div> where</div>
<div><br>
</div>
<div>to get the stack trace when it crashes, or 'bt' on lldb.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Do you have any ideas as to why this larger matrix case should fail when using shift-invert but succeed when using
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
-eps_largest_real ? The fact that the program works and produces correct results <br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
when using the -eps_largest_real option suggests that there is probably nothing wrong with the specification
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
of the problem or the matrices ? It is strange how there is no error message from slepc / Petsc ... the
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
only idea I have at the moment is that perhaps max memory has been exceeded, which could cause such a sudden
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
shutdown? For your reference when running the large matrix case with the -eps_largest_real option I am using
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
about 36 GB of the 148GB available on this machine - does the shift invert approach require substantially
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
more memory for example ?<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I would be very grateful if you have any suggestions to resolve this issue or even ways to clarify it further,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
the performance I have seen with the shift-invert for the small matrix is so impressive it would be great to</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
get that working for the full-size problem.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div><br>
</div>
<div> Many thanks and best wishes,</div>
<div> Dan.<br>
</div>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div id="x_x_gmail-m_3253114932700099937x_gmail-m_-6879576846774134416divRplyFwdMsg" dir="ltr">
<font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>><br>
<b>Sent:</b> Thursday, August 19, 2021 7:58 AM<br>
<b>To:</b> dazza simplythebest <<a href="mailto:sayosale@hotmail.com" target="_blank">sayosale@hotmail.com</a>><br>
<b>Cc:</b> PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Improving efficiency of slepc usage</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>In A) convergence may be slow, especially if the wanted eigenvalues have small magnitude. I would not say 600 iterations is a lot, you probably need many more. In most cases, approach B) is better because it improves convergence of eigenvalues close to
the target, but it requires prior knowledge of your spectrum distribution in order to choose an appropriate target.<br>
<br>
In B) what do you mean that it crashes. If you get an error about factorization, it means that your A-matrix is singular, In that case, try using a nonzero target -eps_target 0.1<br>
<br>
Jose<br>
<br>
<br>
> El 19 ago 2021, a las 7:12, dazza simplythebest <<a href="mailto:sayosale@hotmail.com" target="_blank">sayosale@hotmail.com</a>> escribió:<br>
> <br>
> Dear All,<br>
> I am planning on using slepc to do a large number of eigenvalue calculations<br>
> of a generalized eigenvalue problem, called from a program written in fortran using MPI.<br>
> Thus far I have successfully installed the slepc/PETSc software, both locally and on a cluster,<br>
> and on smaller test problems everything is working well; the matrices are efficiently and
<br>
> correctly constructed and slepc returns the correct spectrum. I am just now starting to move<br>
> towards now solving the full-size 'production run' problems, and would appreciate some
<br>
> general advice on how to improve the solver's performance.<br>
> <br>
> In particular, I am currently trying to solve the problem Ax = lambda Bx whose matrices
<br>
> are of size 50000 (this is the smallest 'production run' problem I will be tackling), and are
<br>
> complex, non-Hermitian. In most cases I aim to find the eigenvalues with the largest real part,
<br>
> although in other cases I will also be interested in finding the eigenvalues whose real part
<br>
> is close to zero.<br>
> <br>
> A)<br>
> Calling slepc 's EPS solver with the following options:<br>
> <br>
> -eps_nev 10 -log_view -eps_view -eps_max_it 600 -eps_ncv 140 -eps_tol 5.0e-6 -eps_largest_real -eps_monitor :monitor_output.txt<br>
> <br>
> <br>
> led to the code successfully running, but failing to find any eigenvalues within the maximum 600 iterations
<br>
> (examining the monitor output it did appear to be very slowly approaching convergence).<br>
> <br>
> B)<br>
> On the same problem I have also tried a shift-invert transformation using the options<br>
> <br>
> -eps_nev 10 -eps_ncv 140 -eps_target 0.0+0.0i -st_type sinvert<br>
> <br>
> -in this case the code crashed at the point it tried to call slepc, so perhaps I have incorrectly specified these options ?<br>
> <br>
> <br>
> Does anyone have any suggestions as to how to improve this performance ( or find out more about the problem) ?<br>
> In the case of A) I can see from watching the slepc videos that increasing ncv
<br>
> may help, but I am wondering , since 600 is a large number of iterations, whether there
<br>
> maybe something else going on - e.g. perhaps some alternative preconditioner may help ?<br>
> In the case of B), I guess there must be some mistake in these command line options?<br>
> Again, any advice will be greatly appreciated.<br>
> Best wishes, Dan.<br>
<br>
</div>
</span></font></div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>