<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sat, Apr 25, 2015 at 8:23 PM, Danyang Su <span dir="ltr"><<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
On 15-04-25 06:03 PM, Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
If this is what you got in your last run<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
at ../../gas_advection/velocity_g.F90:1344<br>
1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316<br>
</blockquote>
then it is still catching floating point underflow, which we do not want. This means either the change I suggested you make in the fp.c code didn't work or it actually uses a different floating point trap than that one. BTW: absurd numbers like 1.0d-316 are often a symptom of uninitialized data; could that be a problem that diff is not filled correctly for all the ic, idim you are using?<br>
<br>
This going round and round is very frustrating and a waste of time. You need to be more proactive yourself and explore the code and poke around to figure out how to solve the problem.<br>
<br>
Please email $PETSC_DIR/$PETSC_ARCH/include/petscvariables.h so I can see what FP trap is being used on your machine.<br>
<br>
Barry<br>
</blockquote>
Do you mean $PETSC_DIR/$PETSC_ARCH/conf/petscvariables? Otherwise I cannot find this file.<br></blockquote><div><br></div><div>Its include/petscconf.h</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Do I need to reconfigure PETSc after changing the code you mentioned?<br></blockquote><div><br></div><div>No, but you need to rebuild.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Danyang<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Apr 25, 2015, at 2:24 PM, Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>> wrote:<br>
<br>
<br>
<br>
On 15-04-25 11:55 AM, Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Apr 25, 2015, at 1:51 PM, Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br>
wrote:<br>
<br>
<br>
<br>
On 15-04-25 11:32 AM, Barry Smith wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I told you this yesterday.<br>
<br>
It is probably stopping here on a harmless underflow. You need to edit the PETSc code to not worry about underflow.<br>
<br>
Edit the file /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c and locate<br>
<br>
#elif defined PETSC_HAVE_XMMINTRIN_H<br>
_MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT);<br>
#else<br>
<br>
change it to<br>
<br>
#elif defined PETSC_HAVE_XMMINTRIN_H<br>
_MM_SET_EXCEPTION_MASK(_MM_MASK_INEXACT | _MM_MASK_UNDERFLOW);<br>
#else<br>
<br>
Then run make gnumake in the PETSc directory to compile the new version. Now link and run the program again with -fp_trap and see where it gets stuck this time.<br>
<br>
Did you do this?<br>
<br>
Barry<br>
<br>
</blockquote>
Yes, I did change the code in fp.c and run 'make gnumake' in the PETSc directory. I just did a double check and ran make gnumake again and got the following information this time.<br>
<br>
<br>
dsu@nwmop:~/Soft/PETSc/petsc-3.5.2$<br>
make gnumake<br>
Building PETSc using GNU Make with 10 build threads<br>
==========================================<br>
make[1]: Entering directory `/home/dsu/Soft/PETSc/petsc-3.5.2'<br>
make[1]: Nothing to be done for `all'.<br>
make[1]: Leaving directory `/home/dsu/Soft/PETSc/petsc-3.5.2'<br>
=========================================<br>
<br>
<br>
Then I recompiled the codes, ran with -fp_trap and still got the following error<br>
<br>
Backtrace for this error:<br>
Note: The EXACT line numbers in the stack are not available,<br>
[2]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[2]PETSC ERROR: is given.<br>
[2]PETSC ERROR: [2] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c<br>
INSTEAD the line number of the start of the function<br>
[3]PETSC ERROR: is given.<br>
[3]PETSC ERROR: [3] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c<br>
[2]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error<br>
[3]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error<br>
<br>
<br>
</blockquote>
This is different then what you sent a few minutes ago where it crashed in hypre.<br>
<br>
Anyways you need to use the -start_in_debugger business I sent in the previous email to see the exact place the problem occurs.<br>
<br>
</blockquote>
Here is the information shown on gdb screen<br>
<br>
Program received signal SIGFPE, Arithmetic exception.<br>
0x00000000006c2bef in velocity_g (l_sufx=1, suffix=..., nmax=12, njamxc=34,<br>
cinfradx=..., radial_coordx=.FALSE., _suffix=3)<br>
at ../../gas_advection/velocity_g.F90:1344<br>
1344 cinfrt = cinfrt_dg(i1) * diff(ic,idim) !diff is a very small value, e.g., 1.0d-316<br>
(gdb)<br>
<br>
After type cont on gdb screen, I got error information as below<br>
<br>
[1]PETSC ERROR: *** unknown floating point error occurred ***<br>
[1]PETSC ERROR: The specific exception can be determined by running in a debugger. When the<br>
[1]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d)<br>
[1]PETSC ERROR: where the result is a bitwise OR of the following flags:<br>
[1]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20<br>
[1]PETSC ERROR: Try option -start_in_debugger<br>
[1]PETSC ERROR: likely location of problem given in stack below<br>
[1]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[1]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[1]PETSC ERROR: is given.<br>
[1]PETSC ERROR: [1] PetscDefaultFPTrap line 379 /home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c<br>
[1]PETSC ERROR: User provided function() line 0 in Unknown file trapped floating point error<br>
[0]PETSC ERROR: *** unknown floating point error occurred ***<br>
[0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the<br>
[0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d)<br>
[0]PETSC ERROR: where the result is a bitwise OR of the following flags:<br>
[0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20<br>
[0]PETSC ERROR: Try option -start_in_debugger<br>
[0]PETSC ERROR: likely location of problem given in stack below<br>
<br>
Thanks,<br>
<br>
Danyang<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks,<br>
<br>
Danyang<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Apr 25, 2015, at 1:05 AM, Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br>
wrote:<br>
<br>
Hi Barry and Satish,<br>
<br>
How can I get rid of unknown floating point error when a very small value is multiplied.<br>
<br>
e.g.,<br>
cinfrt_dg(i1) and diff(ic,idim) are 1.0250235986806329E-008 8.6178408169776945E-317 respectively,<br>
<br>
cinfrt = cinfrt_dg(i1) * diff(ic,idim)<br>
<br>
I get the following error when run with "-fp_trap -start_in_debugger".<br>
<br>
Backtrace for this error:<br>
*** unknown floating point error occurred ***<br>
[2]PETSC ERROR: The specific exception can be determined by running in a debugger. When the<br>
[2]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3d)<br>
[2]PETSC ERROR: cinfrt_dg(i1),diff(ic,idim) 1.0250235986806329E-008 8.6178408169776945E-317<br>
where the result is a bitwise OR of the following flags:<br>
[2]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 FE_UNDERFLOW=0x10 FE_INEXACT=0x20<br>
[2]PETSC ERROR: Try option -start_in_debugger<br>
[2]PETSC ERROR: likely location of problem given in stack below<br>
<br>
Thanks,<br>
<br>
Danyang<br>
<br>
On 15-04-24 01:54 PM, Danyang Su wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 15-04-24 01:23 PM, Satish Balay wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
c 4 1.0976214263087059E-067<br>
<br>
I don't think this number can be stored in a real*4.<br>
<br>
Satish<br>
<br>
</blockquote>
Thanks, Satish. It is caused by this number.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Fri, 24 Apr 2015, Danyang Su wrote:<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 15-04-24 11:12 AM, Barry Smith wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Apr 24, 2015, at 1:05 PM, Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br>
wrote:<br>
<br>
Hi All,<br>
<br>
One of my case crashes because of floating point exception when using 4<br>
processors, as shown below. But if I run this case with 1 processor, it<br>
works fine. I have tested the codes with around 100 cases up to 768<br>
processors, all other cases work fine. I just wonder if this kind of error<br>
is caused because of NaN in jacobi matrix, RHS or preconditioner?<br>
<br>
</blockquote>
Yes, almost for sure it is one of these places.<br>
<br>
First run the bad case with -fp_trap if all goes well you'll see the<br>
function where the FPE is generated. Then run also with -start_in_debugger<br>
and<br>
type cont in all four debugger windows. When the FPE happens the debugger<br>
should stop showing exactly where the FPE happens.<br>
<br>
Barry<br>
<br>
</blockquote>
Hi Barry,<br>
<br>
If run with -fp_trap -start_in_debugger, I got the following error<br>
<br>
[0]PETSC ERROR: *** unknown floating point error occurred ***<br>
[0]PETSC ERROR: The specific exception can be determined by running in a<br>
debugger. When the<br>
[0]PETSC ERROR: debugger traps the signal, the exception can be found with<br>
fetestexcept(0x3d)<br>
[0]PETSC ERROR: where the result is a bitwise OR of the following flags:<br>
[0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8<br>
FE_UNDERFLOW=0x10 FE_INEXACT=0x20<br>
[0]PETSC ERROR: Try option -start_in_debugger<br>
[0]PETSC ERROR: likely location of problem given in stack below<br>
[0]PETSC ERROR: --------------------- Stack Frames<br>
------------------------------------<br>
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[0]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[0]PETSC ERROR: is given.<br>
[0]PETSC ERROR: [0] PetscDefaultFPTrap line 379<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/src/sys/error/fp.c<br>
[0]PETSC ERROR: User provided function() line 0 in Unknown file trapped<br>
floating point error<br>
<br>
Program received signal SIGABRT: Process abort signal.<br>
<br>
Backtrace for this error:<br>
#0 0x7F4FEAB1C7D7<br>
#1 0x7F4FEAB1CDDE<br>
#2 0x7F4FE9E1AD3F<br>
#3 0x7F4FE9E1ACC9<br>
#4 0x7F4FE9E1E0D7<br>
#5 0x7F4FEB0B6DCB<br>
#6 0x7F4FEB0B1825<br>
#7 0x7F4FEB0B817F<br>
#8 0x7F4FE9E1AD3F<br>
#9 0x6972C8 in tprfrtlc_ at tprfrtlc.F90:2393 (discriminator 3)<br>
#10 0x4C6C87 in gcreact_ at gcreact.F90:678<br>
#11 0x707E19 in initicrt_ at initicrt.F90:589<br>
#12 0x4F42D0 in initprob_ at initprob.F90:430<br>
#13 0x5AAF72 in driver_pc at driver_pc.F90:438<br>
<br>
I checked the code at tprfrtlc.F90:2393,<br>
<br>
realbuffer_gb(1:nvars) = (/time,(c(ic),ic=1,nc-1), &<br>
(cx(ix),ix=1,nxout)/)<br>
<br>
All the values (time, c, cx) are reasonable, as shown below. The only<br>
possibility is that realbuffer_gb is in declared as real*4 if using sing<br>
precision output while time, c, cx are declared in real*8. I have a lot of<br>
similar data conversion from real*8 to real*4 output, other code does not<br>
return error.<br>
<br>
time 0.0000000000000000<br>
c 1 9.9999999999999995E-008<br>
c 2 3.1555251077549618E-003<br>
c 3 7.1657814842179362E-008<br>
c 4 1.0976214263087059E-067<br>
c 5 5.2879822292305797E-004<br>
c 6 9.9999999999999964E-005<br>
c 7 6.4055731968811337E-005<br>
c 8 3.4607572892578404E-020<br>
cx 1 3.4376650636008101E-005<br>
cx 2 7.3989678854017763E-012<br>
cx 3 9.5317170613607207E-012<br>
cx 4 2.2344525794718353E-015<br>
cx 5 3.0624685689695889E-008<br>
cx 6 1.0046157902783967E-007<br>
cx 7 1.5320169154914984E-004<br>
cx 8 8.6930292776346176E-014<br>
cx 9 3.5944267559348721E-005<br>
cx 10 3.0072645866951157E-018<br>
cx 11 2.3592486321095017E-013<br>
<br>
Thanks,<br>
<br>
Danyang<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I can check all the entries of jacobi matrix to see if the value is valid,<br>
but this seems not a good idea as it takes a long time to reach this<br>
point. If I restart the simulation from a specified time (e.g., 7.685 in<br>
this case), then the error does not occur.<br>
<br>
Would you please give me any suggestion on debugging this case?<br>
<br>
Thanks and Regards,<br>
<br>
Danyang<br>
<br>
<br>
timestep: 2730 time: 7.665E+00 years delt: 1.000E-02 years iter: 1<br>
timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00<br>
timestep: 2731 time: 7.675E+00 years delt: 1.000E-02 years iter: 1<br>
timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00<br>
timestep: 2732 time: 7.685E+00 years delt: 1.000E-02 years iter: 1<br>
timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00<br>
timestep: 2733 time: 7.695E+00 years delt: 1.000E-02 years iter: 1<br>
timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00<br>
timestep: 2734 time: 7.705E+00 years delt: 1.000E-02 years iter: 1<br>
timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00<br>
Reduce time step for reactive transport<br>
timestep: 2734 time: 7.700E+00 years delt: 5.000E-03 years iter: 1<br>
timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00<br>
Reduce time step for reactive transport<br>
timestep: 2734 time: 7.697E+00 years delt: 2.500E-03 years iter: 1<br>
timestep: max.sia: 0.000E+00 tol.sia: 0.000E+00<br>
[1]PETSC ERROR: --------------------- Error Message<br>
--------------------------------------------------------------<br>
[1]PETSC ERROR: Floating point exception<br>
[2]PETSC ERROR: --------------------- Error Message<br>
--------------------------------------------------------------<br>
[2]PETSC ERROR: Floating point exception<br>
[2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite<br>
at end of function: Parameter number 3<br>
[2]PETSC ERROR: See<br>
<a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>
<br>
for trouble shooting.<br>
[2]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014<br>
[2]PETSC ERROR: [1]PETSC ERROR: Vec entry at local location 0 is<br>
not-a-number or infinite at end of function: Parameter number 3<br>
[1]PETSC ERROR: See<br>
<a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>
<br>
for trouble shooting.<br>
[1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014<br>
[1]PETSC ERROR: ../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by<br>
dsu Thu Apr 23 15:38:52 2015<br>
[1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc<br>
--with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich<br>
--download-mumps --download-hypre --download-superlu_dist --download-metis<br>
--download-parmetis --download-scalapack<br>
[1]PETSC ERROR: #1 VecValidValues() line 34 in<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c<br>
../min3p_thcm_petsc_dbg on a linux-gnu-dbg named nwmop by dsu Thu Apr 23<br>
15:38:52 2015<br>
[2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-dbg --with-cc=gcc<br>
--with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich<br>
--download-mumps --download-hypre --download-superlu_dist --download-metis<br>
--download-parmetis --download-scalapack<br>
[2]PETSC ERROR: #1 VecValidValues() line 34 in<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/src/vec/vec/interface/rvector.c<br>
[2]PETSC ERROR: [1]PETSC ERROR: #2 PCApply() line 442 in<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c<br>
[1]PETSC ERROR: #2 PCApply() line 442 in<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/pc/interface/precon.c<br>
[2]PETSC ERROR: #3 KSP_PCApply() line 230 in<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h<br>
#3 KSP_PCApply() line 230 in<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/include/petsc-private/kspimpl.h<br>
[1]PETSC ERROR: #4 KSPInitialResidual() line 63 in<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c<br>
[2]PETSC ERROR: #4 KSPInitialResidual() line 63 in<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itres.c<br>
[1]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c<br>
[2]PETSC ERROR: #5 KSPSolve_GMRES() line 234 in<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/impls/gmres/gmres.c<br>
[2]PETSC ERROR: #6 KSPSolve() line 459 in<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c<br>
[1]PETSC ERROR: #6 KSPSolve() line 459 in<br>
/home/dsu/Soft/PETSc/petsc-3.5.2/src/ksp/ksp/interface/itfunc.c<br>
^C[mpiexec@nwmop] Sending Ctrl-C to processes as requested<br>
[mpiexec@nwmop] Press Ctrl-C again to force abort<br>
<br>
</blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>