[Nek5000-users] proj_ortho error in pipe flow

nek5000-users at lists.mcs.anl.gov nek5000-users at lists.mcs.anl.gov
Fri Apr 7 10:18:23 CDT 2017


Dear Nek users,

I am running a simple pipe flow on a Cray XC40 (Hazel Hen). After 32 
steps, I get this problem (see attachment for complete logfile):

Step     32, t= 3.2000000E-02, DT= 1.0000000E-03, C=  0.025 3.2660E-01 
1.0526E-02
              Solving for fluid
          32  Project velx           6.6444E-09   2.4759E-09 
2.6836E+00   8   0
          32  Hmholtz velx       1   3.9238E-09   6.7918E-07 1.0000E-08
          32  Project vely           6.8730E-09   2.1046E-09 
3.2656E+00   8   0
          32  Hmholtz vely       1   3.6166E-09   5.9974E-07 1.0000E-08
          32  Project velz           5.2396E-03   2.5121E-06 
2.0858E+03   8   0
          32  Hmholtz velz       3   7.6028E-10   9.3795E-04 1.0000E-08
        32proj_ortho:    2   8 velz   Detect rank deficiency: NaN  
1.0000E+00
        32proj_ortho:    1   8 velz   Detect rank deficiency: NaN  
1.0000E+00
          32  U-PRES gmres      29   9.9376E-09   5.2213E-04 
1.0000E-08   8.7926E-03   1.7780E-02
           32  DNORM, DIVEX  2.704668739973178E-005 9.937641774554489E-009
       32 0.3200000E-01  1.61568E-01  6.88948E-05  7.85330E-01 
7.85399E-01 volflow Z
          32  Fluid done  3.2000E-02  2.4042E-02

In the next step I get CFL, CTarg! and the simulation exits.

This problem does neither show up on my local machine nor on another 
smaller cluster. Moreover, when I set either parameter 94 or 95 to 0, I 
do not see this error. It only appears when p94 and p95 are both unequal 
0. In my case I set them to 9.

Do you know what might be causing this issue and how to solve it for the 
Cray XC40 system?


Best Regards
Steffen
-------------- next part --------------
Starting at Fri Apr  7 16:47:45 CEST 2017
Hosts: eslogin002.hww.de
Nodes: 1
Processes per node: 24
Cores: 24
------------------
/----------------------------------------------------------\\
|      _   __ ______ __ __  ______  ____   ____   ____     |
|     / | / // ____// //_/ / ____/ / __ \\/ __ \\/ __ \\   |
|    /  |/ // __/  / ,<   /___ \\ / / / // / / // / / /    |
|   / /|  // /___ / /| | ____/ / / /_/ // /_/ // /_/ /     |
|  /_/ |_//_____//_/ |_|/_____/  \\___/ \\___/ \\___/      |
|                                                          |
|----------------------------------------------------------|
|                                                          |
| NEK5000:  Open Source Spectral Element Solver            |
| COPYRIGHT (c) 2008-2017 UCHICAGO ARGONNE, LLC            |
| Version:  17.0.0                                         |
| Web:      http://nek5000.mcs.anl.gov                     |
|                                                          |
\\----------------------------------------------------------/
                                                              

 Number of processors:          24
 REAL    wdsize      :           8
 INTEGER wdsize      :           4
 Timer accuracy      : 9.54E-08
 Reading /lustre/cray/ws8/ws/xxxxx-nek_tests/nek_runs/pipe/hydro/run/hydro.rea                                                            

 mapping elements to processors
 Reading /lustre/cray/ws8/ws/xxxxx-nek_tests/nek_runs/pipe/hydro/run/hydro.map                                                            
           0          32          32         768         768  NELV
           1          32          32         768         768  NELV
           5          32          32         768         768  NELV
           8          32          32         768         768  NELV
           2          32          32         768         768  NELV
           9          32          32         768         768  NELV
          10          32          32         768         768  NELV
           4          32          32         768         768  NELV
          11          32          32         768         768  NELV
           3          32          32         768         768  NELV
          18          32          32         768         768  NELV
          12          32          32         768         768  NELV
          20          32          32         768         768  NELV
           7          32          32         768         768  NELV
           6          32          32         768         768  NELV
          23          32          32         768         768  NELV
          22          32          32         768         768  NELV
          21          32          32         768         768  NELV
          19          32          32         768         768  NELV
          15          32          32         768         768  NELV
          16          32          32         768         768  NELV
          14          32          32         768         768  NELV
          17          32          32         768         768  NELV
          13          32          32         768         768  NELV
 RANK     0 IEG     108     112     117     118     119     120     300     304
                    305     306     309     310     311     312     492     494
                    496     497     498     501     502     503     504     684
                    686     688     689     690     693     694     695     696
 element load imbalance:            0          32          32
 done :: mapping elements to processors
  
           0  objects found
 118   Parameters from file:
   1      1.00000     P001: DENSITY
   2     -5300.00     P002: VISCOSITY
   7      1.00000     P007: RHOCP
   8      1.00000     P008: CONDUCT
  11      103.000     P011: NSTEPS
  12     -1.00000E-03 P012: DT
  13      20.0000     P013: IOCOMM
  15      10.0000     P015: IOSTEP
  21      1.00000E-08 P021: DIVERGENCE
  22      1.00000E-08 P022: HELMHOLTZ
  24      1.00000E-02 P024: TOLREL
  25      1.00000E-02 P025: TOLABS
  26      0.50000     P026: COURANT/NTAU
  27      3.00000     P027: TORDER
  54     -3.00000     P054: fixed flow rate dir: |p54|=1,2,3=x,y,z
  55      1.00000     P055: vol.flow rate (p54>0) or Ubar (p54<0)
  59      0.00000     P059: !=0 --> full Jac. eval. for each el.
  66      6.00000     P066: output : <0=ascii, else binary
  67      6.00000     P067: restart: <0=ascii, else binary
  68      10.0000     P068: STAT_COMP: how often you compute stats
  69      50.0000     P069: STAT_OUTP: how often you write stats
  70      50.0000     P070: CHKPTSTP: how often you write restart files (rs8)
  93      20.0000     P093: Number of previous pressure solns saved
  94      9.00000     P094: start projecting velocity after p94 step
  95      9.00000     P095: start projecting pressure after p95 step
  99      4.00000     P099: dealiasing: <0--> off /3--> old /4-->new
 102      1.00000     P102: Dump out divergence at each time step
 103      0.01000     P103: weight of stabilizing filter
  
 nelgt/nelgv/lelt:         768         768        1200
 lx1  /lx2  /lx3 :           5           3           5
 done :: read .rea file   0.25736E-01 sec

 setup mesh topology
   Right-handed check complete for     768 elements. OK.
   setvert3d:   5       29456       50192       29456       29456
 call usrsetvert
 done :: usrsetvert

gs_setup: 12560 unique labels shared
   pairwise times (avg, min, max): 1.80612e-05 1.59979e-05 2.01941e-05
   crystal router                : 1.46588e-05 1.43051e-05 1.49012e-05
   all reduce                    : 0.000158255 0.000156999 0.000159812
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 223072 152956 276732
   buffer bytes (avg, min, max): 37301.3 25760 47008
   setupds time 8.6558E-03 seconds   0  5       29456         768
           8  max multiplicity
 done :: setup mesh topology
  
 call usrdat
 done :: usrdat

 generate geometry data
 NOTE: All elements deformed , param(59) ^=0
 done :: generate geometry data
  
 call usrdat2
 done :: usrdat2

 regenerate geometry data           1
 NOTE: All elements deformed , param(59) ^=0
 done :: regenerate geometry data           1
  
 verify mesh topology
 -0.500000000000000       0.500000000000000       Xrange
 -0.500000000000000       0.500000000000000       Yrange
  0.000000000000000E+000   1.00000000000000       Zrange
 done :: verify mesh topology
  
 IFTRAN   = T
 IFFLOW   = T
 IFHEAT   = F
 IFSPLIT  = F
 IFLOMACH = F
 IFUSERVP = F
 IFUSERMV = F
 IFSTRS   = F
 IFCHAR   = F
 IFCYCLIC = F
 IFAXIS   = F
 IFMVBD   = F
 IFMELT   = F
 IFMODEL  = F
 IFKEPS   = F
 IFMOAB   = F
 IFNEKNEK = F
 IFSYNC   = T
   
 IFVCOR   = T
 IFINTQ   = F
 IFCWUZ   = F
 IFSWALL  = F
 IFGEOM   = F
 IFSURT   = F
 IFWCNO   = F
   
 IFTMSH for field           1    =  F
 IFADVC for field           1    =  T
 IFNONL for field           1    =  F
   
 Dealiasing enabled, lxd=           8
  
 Estimated eigenvalues
 EIGAA =    9.86960440108936     
 EIGGA =    450284.517077610     
 EIGAE =    9.86960440108936     
 EIGAS =   8.333333333333333E-002
 EIGGE =    450284.517077610     
 EIGGS =    2.00000000000000     
  
 verify mesh topology
 -0.500000000000000       0.500000000000000       Xrange
 -0.500000000000000       0.500000000000000       Yrange
  0.000000000000000E+000   1.00000000000000       Zrange
 done :: verify mesh topology
  
  E-solver strategy:  1 itr
 mg_nx:           1           2           4
 mg_ny:           1           2           4
 mg_nz:           1           2           4
 call usrsetvert
 done :: usrsetvert

gs_setup: 680 unique labels shared
   pairwise times (avg, min, max): 1.29193e-05 1.18017e-05 1.37091e-05
   crystal router                : 5.6078e-06 5.48363e-06 5.72205e-06
   all reduce                    : 1.7484e-05 1.73092e-05 1.75953e-05
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 16252 12052 19156
   buffer bytes (avg, min, max): 2465.33 1904 2944
   setupds time 1.4441E-03 seconds   1  2         836         768
   setvert3d:   4       15180       21324       15180       15180
 call usrsetvert
 done :: usrsetvert

gs_setup: 6964 unique labels shared
   pairwise times (avg, min, max): 1.5373e-05 1.35899e-05 1.76907e-05
   crystal router                : 1.00841e-05 9.799e-06 1.02997e-05
   all reduce                    : 8.25336e-05 8.18014e-05 8.35896e-05
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 127296 87988 156980
   buffer bytes (avg, min, max): 21137.3 14768 26496
   setupds time 4.7832E-03 seconds   2  4       15180         768
   setvert3d:   3        5640        6408        5640        5640
 call usrsetvert
 done :: usrsetvert

gs_setup: 3004 unique labels shared
   pairwise times (avg, min, max): 1.4472e-05 1.28031e-05 1.63078e-05
   crystal router                : 6.6479e-06 6.50883e-06 6.69956e-06
   all reduce                    : 3.99897e-05 3.96013e-05 4.05073e-05
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 58356 41020 71068
   buffer bytes (avg, min, max): 9525.33 6816 11808
   setupds time 2.9030E-03 seconds   3  3        5640         768
   setvert3d:   5       29456       50192       29456       29456
 call usrsetvert
 done :: usrsetvert

gs_setup: 12560 unique labels shared
   pairwise times (avg, min, max): 1.54346e-05 1.33038e-05 1.74046e-05
   crystal router                : 1.46826e-05 1.44005e-05 1.49012e-05
   all reduce                    : 0.000159989 0.000158787 0.0001616
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 223072 152956 276732
   buffer bytes (avg, min, max): 37301.3 25760 47008
   setupds time 7.5829E-03 seconds   4  5       29456         768
 setup h1 coarse grid, nx_crs=           2
 call usrsetvert
 done :: usrsetvert

gs_setup: 680 unique labels shared
   pairwise times (avg, min, max): 1.23511e-05 1.03951e-05 1.38998e-05
   crystal router                : 4.39882e-06 4.29153e-06 4.48227e-06
   all reduce                    : 1.73748e-05 1.719e-05 1.74999e-05
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 16252 12052 19156
   buffer bytes (avg, min, max): 2465.33 1904 2944
 done :: setup h1 coarse grid   5.075931549072266E-003  sec
  
 call usrdat3
 done :: usrdat3

 set initial conditions
 nekuic (1) for ifld            1
 call nekuic for vel  
 xyz min   -0.50000     -0.50000       0.0000    
 uvwpt min   0.0000       0.0000       1.0000       0.0000       0.0000    
 PS min     0.99000E+22
 xyz max    0.50000      0.50000       1.0000    
 uvwpt max   0.0000       0.0000       1.0000       0.0000       0.0000    
 PS max    -0.99000E+22
 done :: set initial conditions
  
 call userchk

        0  0.0000E+00 Write checkpoint
       FILE:
 hydro0.f00001                                                                  
                                                      

        0  0.0000E+00 done :: Write checkpoint
                              file size =      3.0    MB
                              avg data-throughput =    83.0MB/s
                              io-nodes =    24

 done :: userchk

gridpoints unique/tot:         50192        96000
  dofs:                 48144                 20736
  
 Initialization successfully completed   0.11032     sec

Starting time loop ...

     DT/DTCFL/DTFS/DTINIT   0.100E-02   0.000E+00   0.593-322   0.100E-02
Step      1, t= 1.0000000E-03, DT= 1.0000000E-03, C=  0.023 0.0000E+00 0.0000E+00
             Solving for fluid
          1  Hmholtz VELZ       1   1.7018E+00   1.8868E-04   1.0000E-08   1.0000E+03   F
          1  Hmholtz VELZ       2   1.3450E-02   1.8868E-04   1.0000E-08   1.0000E+03   F
          1  Hmholtz VELZ       3   1.5849E-04   1.8868E-04   1.0000E-08   1.0000E+03   F
          1  Hmholtz VELZ       4   1.6074E-06   1.8868E-04   1.0000E-08   1.0000E+03   F
          1  Hmholtz VELZ       5   2.2425E-08   1.8868E-04   1.0000E-08   1.0000E+03   F
          1  Hmholtz VELZ       6   4.0630E-10   1.8868E-04   1.0000E-08   1.0000E+03   F
          1  Hmholtz VELZ       5   4.0630E-10   1.7018E+00   1.0000E-08
    1 1.00000E-08 2.17259E-13 4.01456E-13 5.41176E-01       1 Divergence
          1  U-PRES gmres       1   2.1726E-13   4.0146E-13   1.0000E-08   7.1907E-04   1.1599E-03
           1  DNORM, DIVEX  2.198172942125789E-013  2.172586674725858E-013
          1  Hmholtz VELZ       1   9.9774E-01   1.8868E-04   1.0000E-08   1.0000E+03   F
          1  Hmholtz VELZ       2   6.1199E-03   1.8868E-04   1.0000E-08   1.0000E+03   F
          1  Hmholtz VELZ       3   7.2730E-05   1.8868E-04   1.0000E-08   1.0000E+03   F
          1  Hmholtz VELZ       4   1.4029E-06   1.8868E-04   1.0000E-08   1.0000E+03   F
          1  Hmholtz VELZ       5   1.9624E-08   1.8868E-04   1.0000E-08   1.0000E+03   F
          1  Hmholtz VELZ       6   3.5717E-10   1.8868E-04   1.0000E-08   1.0000E+03   F
          1  Hmholtz VELZ       5   3.5717E-10   9.9774E-01   1.0000E-08
    1 1.00000E-04 2.83402E-13 4.08036E-13 6.94550E-01       0 Divergence
          0  U-PRES gmres       1   2.8340E-13   4.0804E-13   1.0000E-04   3.6812E-04   7.2598E-04
        1  7.81677E-04  1.00000E+00  1.00000E+00 basflow Z
       1 0.1000000E-02  4.76073E+00  3.72135E-03  7.81677E-01  7.85399E-01 volflow Z
          1  Fluid done  1.0000E-03  7.6170E-03
filt amp 0.0000 0.0000 0.0000 0.0000 0.0100
filt trn 1.0000 1.0000 1.0000 1.0000 0.9900
Step      2, t= 2.0000000E-03, DT= 1.0000000E-03, C=  0.023 1.2206E-02 1.2206E-02
             Solving for fluid
          2  Hmholtz VELX       0   9.4271E-12   9.4271E-12   1.0000E-08
          2  Hmholtz VELY       0   9.4227E-12   9.4227E-12   1.0000E-08
          2  Hmholtz VELZ       5   2.4847E-10   3.2933E+00   1.0000E-08
          2  U-PRES gmres       1   9.4610E-13   1.5322E-12   1.0000E-08   4.5395E-04   7.9298E-04
           2  DNORM, DIVEX  9.464175787172753E-013  9.461022252443596E-013
          2  Hmholtz VELZ       4   4.0433E-09   9.9774E-01   1.0000E-08
          0  U-PRES gmres       1   1.9253E-12   3.1157E-12   1.0000E-04   3.5119E-04   6.8712E-04
        2  5.21158E-04  1.00000E+00  1.00000E+00 basflow Z
       2 0.2000000E-02 -2.03310E+00 -1.05957E-03  7.86458E-01  7.85399E-01 volflow Z
          2  Fluid done  2.0000E-03  6.4468E-03
Step      3, t= 3.0000000E-03, DT= 1.0000000E-03, C=  0.023 2.2941E-02 1.0735E-02
             Solving for fluid
          3  Hmholtz VELX       0   9.1600E-11   9.1600E-11   1.0000E-08
          3  Hmholtz VELY       0   9.1590E-11   9.1590E-11   1.0000E-08
          3  Hmholtz VELZ       5   1.0894E-10   3.6725E+00   1.0000E-08
          3  U-PRES gmres       1   2.9812E-12   4.7110E-12   1.0000E-08   4.4107E-04   7.8297E-04
           3  DNORM, DIVEX  2.981441029198212E-012  2.981214196025064E-012
          3  Hmholtz VELZ       4   1.8402E-09   9.9774E-01   1.0000E-08
          0  U-PRES gmres       1   7.1775E-13   1.1612E-12   1.0000E-04   3.5119E-04   7.0310E-04
        3  4.26414E-04  1.00000E+00  1.00000E+00 basflow Z
       3 0.3000000E-02  1.73615E+00  7.40317E-04  7.84658E-01  7.85399E-01 volflow Z
          3  Fluid done  3.0000E-03  6.2871E-03
Step      4, t= 4.0000000E-03, DT= 1.0000000E-03, C=  0.024 3.1733E-02 8.7922E-03
             Solving for fluid
          4  Hmholtz VELX       0   8.4520E-11   8.4520E-11   1.0000E-08
          4  Hmholtz VELY       0   8.4533E-11   8.4533E-11   1.0000E-08
          4  Hmholtz VELZ       4   5.7268E-09   3.2214E+00   1.0000E-08
          4  U-PRES gmres       1   4.0800E-12   7.5439E-12   1.0000E-08   4.3988E-04   7.8201E-04
           4  DNORM, DIVEX  4.080101052278270E-012  4.079985609915090E-012
       4 0.4000000E-02  2.23876E-01  9.54639E-05  7.85303E-01  7.85399E-01 volflow Z
          4  Fluid done  4.0000E-03  3.5310E-03
Step      5, t= 5.0000000E-03, DT= 1.0000000E-03, C=  0.024 3.7650E-02 5.9168E-03
             Solving for fluid
          5  Hmholtz VELX       0   1.1332E-10   1.1332E-10   1.0000E-08
          5  Hmholtz VELY       0   1.1323E-10   1.1323E-10   1.0000E-08
          5  Hmholtz VELZ       4   5.5891E-09   3.1424E+00   1.0000E-08
          5  U-PRES gmres       1   4.0020E-12   6.0607E-12   1.0000E-08   4.5204E-04   7.9513E-04
           5  DNORM, DIVEX  4.002157619691075E-012  4.001992348531620E-012
       5 0.5000000E-02  2.20709E-01  9.41133E-05  7.85304E-01  7.85399E-01 volflow Z
          5  Fluid done  5.0000E-03  3.5372E-03
Step      6, t= 6.0000000E-03, DT= 1.0000000E-03, C=  0.024 4.5496E-02 7.8461E-03
             Solving for fluid
          6  Hmholtz VELX       1   6.1591E-13   1.0781E-10   1.0000E-08
          6  Hmholtz VELY       1   6.1599E-13   1.0766E-10   1.0000E-08
          6  Hmholtz VELZ       4   5.2585E-09   3.0485E+00   1.0000E-08
          6  U-PRES gmres       1   1.0207E-11   1.9025E-11   1.0000E-08   4.5419E-04   8.0085E-04
           6  DNORM, DIVEX  1.020720384931466E-011  1.020715819732652E-011
       6 0.6000000E-02  2.17638E-01  9.28039E-05  7.85306E-01  7.85399E-01 volflow Z
          6  Fluid done  6.0000E-03  3.7589E-03
Step      7, t= 7.0000000E-03, DT= 1.0000000E-03, C=  0.024 5.2164E-02 6.6681E-03
             Solving for fluid
          7  Hmholtz VELX       1   7.4415E-13   1.2102E-10   1.0000E-08
          7  Hmholtz VELY       1   7.5130E-13   1.2107E-10   1.0000E-08
          7  Hmholtz VELZ       4   4.9167E-09   2.9577E+00   1.0000E-08
          7  U-PRES gmres       1   1.4285E-11   2.7817E-11   1.0000E-08   4.3821E-04   7.7796E-04
           7  DNORM, DIVEX  1.428514599120713E-011  1.428505538012936E-011
       7 0.7000000E-02  2.14661E-01  9.15347E-05  7.85307E-01  7.85399E-01 volflow Z
          7  Fluid done  7.0000E-03  3.7742E-03
Step      8, t= 8.0000000E-03, DT= 1.0000000E-03, C=  0.024 5.8715E-02 6.5508E-03
             Solving for fluid
          8  Hmholtz VELX       1   1.4103E-12   1.9137E-10   1.0000E-08
          8  Hmholtz VELY       1   1.4529E-12   1.9315E-10   1.0000E-08
          8  Hmholtz VELZ       4   4.6070E-09   2.8736E+00   1.0000E-08
          8  U-PRES gmres       1   1.7677E-11   3.9040E-11   1.0000E-08   4.4584E-04   7.9203E-04
           8  DNORM, DIVEX  1.767719693310791E-011  1.767720753586405E-011
       8 0.8000000E-02  2.11773E-01  9.03031E-05  7.85308E-01  7.85399E-01 volflow Z
          8  Fluid done  8.0000E-03  3.8049E-03
Step      9, t= 9.0000000E-03, DT= 1.0000000E-03, C=  0.024 6.4911E-02 6.1960E-03
             Solving for fluid
          9  Hmholtz velx       1   1.4859E-12   2.3231E-10   1.0000E-08
          9  Hmholtz vely       1   1.7041E-12   2.4480E-10   1.0000E-08
          9  Hmholtz velz       4   4.3245E-09   2.7952E+00   1.0000E-08
          9  U-PRES gmres       1   2.5635E-11   5.4938E-11   1.0000E-08   4.1890E-04   7.6914E-04
           9  DNORM, DIVEX  2.563482495492407E-011  2.563482607944534E-011
       9 0.9000000E-02  2.08969E-01  8.91071E-05  7.85309E-01  7.85399E-01 volflow Z
          9  Fluid done  9.0000E-03  4.5190E-03
Step     10, t= 1.0000000E-02, DT= 1.0000000E-03, C=  0.024 7.1832E-02 6.9211E-03
             Solving for fluid
         10  Project velx           1.0928E-12   1.0380E-12   1.0528E+00   1   1
         10  Hmholtz velx       1   2.2565E-12   2.9156E-10   1.0000E-08
         10  Project vely           1.1238E-12   1.1064E-12   1.0158E+00   1   1
         10  Hmholtz vely       1   2.4580E-12   3.2426E-10   1.0000E-08
         10  Project velz           8.4125E-03   1.4438E-04   5.8267E+01   1   1
         10  Hmholtz velz       4   1.3860E-10   4.3668E-02   1.0000E-08
         10  U-PRES gmres       1   3.5152E-11   1.0643E-10   1.0000E-08   4.2200E-04   7.5507E-04
          10  DNORM, DIVEX  3.515225402474910E-011  3.515227680759790E-011
      10 0.1000000E-01  2.06244E-01  8.79452E-05  7.85311E-01  7.85399E-01 volflow Z
         10  Fluid done  1.0000E-02  5.6860E-03

       10  1.0000E-02 Write checkpoint
       FILE:
 hydro0.f00002                                                                  
                                                      

       10  1.0000E-02 done :: Write checkpoint
                              file size =      1.9    MB
                              avg data-throughput =    99.6MB/s
                              io-nodes =    24

Step     11, t= 1.1000000E-02, DT= 1.0000000E-03, C=  0.024 9.8775E-02 2.6943E-02
             Solving for fluid
         11  Project velx           2.0887E-12   1.6492E-12   1.2664E+00   2   0
         11  Hmholtz velx       1   3.8614E-12   4.6283E-10   1.0000E-08
         11  Project vely           2.0993E-12   1.8999E-12   1.1050E+00   2   0
         11  Hmholtz vely       1   3.5506E-12   5.0358E-10   1.0000E-08
         11  Project velz           8.1908E-03   2.0834E-06   3.9314E+03   2   0
         11  Hmholtz velz       3   5.3533E-10   8.1921E-04   1.0000E-08
         11  U-PRES gmres       1   5.9757E-11   1.1938E-10   1.0000E-08   4.1008E-04   7.3814E-04
          11  DNORM, DIVEX  5.975704618222100E-011  5.975696721245879E-011
      11 0.1100000E-01  2.03595E-01  8.68158E-05  7.85312E-01  7.85399E-01 volflow Z
         11  Fluid done  1.1000E-02  5.4030E-03
Step     12, t= 1.2000000E-02, DT= 1.0000000E-03, C=  0.024 1.0672E-01 7.9491E-03
             Solving for fluid
         12  Project velx           3.3627E-12   1.2342E-12   2.7247E+00   3   0
         12  Hmholtz velx       1   3.3679E-12   3.8173E-10   1.0000E-08
         12  Project vely           3.3632E-12   1.0434E-12   3.2233E+00   3   0
         12  Hmholtz vely       1   3.5490E-12   3.4661E-10   1.0000E-08
         12  Project velz           7.9792E-03   1.5430E-07   5.1712E+04   3   0
         12  Hmholtz velz       2   2.9327E-09   5.1295E-05   1.0000E-08
         12  U-PRES gmres       1   5.5735E-11   1.9372E-10   1.0000E-08   4.0412E-04   7.3099E-04
          12  DNORM, DIVEX  5.573487507738371E-011  5.573496867795029E-011
      12 0.1200000E-01  2.01021E-01  8.57180E-05  7.85313E-01  7.85399E-01 volflow Z
         12  Fluid done  1.2000E-02  5.4240E-03
Step     13, t= 1.3000000E-02, DT= 1.0000000E-03, C=  0.024 1.1473E-01 8.0049E-03
             Solving for fluid
         13  Project velx           6.4193E-12   2.2025E-12   2.9146E+00   4   0
         13  Hmholtz velx       1   3.8826E-12   5.9537E-10   1.0000E-08
         13  Project vely           6.4350E-12   2.1703E-12   2.9650E+00   4   0
         13  Hmholtz vely       1   4.8476E-12   6.2835E-10   1.0000E-08
         13  Project velz           7.7771E-03   4.1469E-08   1.8754E+05   4   0
         13  Hmholtz velz       2   1.6601E-09   1.5089E-05   1.0000E-08
         13  U-PRES gmres       1   4.3753E-11   1.0105E-10   1.0000E-08   4.0913E-04   7.4196E-04
          13  DNORM, DIVEX  4.375244503850638E-011  4.375262559056995E-011
      13 0.1300000E-01  1.98517E-01  8.46505E-05  7.85314E-01  7.85399E-01 volflow Z
         13  Fluid done  1.3000E-02  5.6980E-03
Step     14, t= 1.4000000E-02, DT= 1.0000000E-03, C=  0.024 1.2315E-01 8.4231E-03
             Solving for fluid
         14  Project velx           2.1362E-12   1.0723E-12   1.9921E+00   5   0
         14  Hmholtz velx       1   3.9810E-12   3.8491E-10   1.0000E-08
         14  Project vely           2.2193E-12   1.2086E-12   1.8363E+00   5   0
         14  Hmholtz vely       1   4.5744E-12   4.3292E-10   1.0000E-08
         14  Project velz           7.5841E-03   7.8784E-10   9.6264E+06   5   0
         14  Hmholtz velz       1   2.6942E-09   2.6278E-07   1.0000E-08
         14  U-PRES gmres       1   3.4870E-11   8.9533E-11   1.0000E-08   4.1699E-04   7.5006E-04
          14  DNORM, DIVEX  3.487049980630507E-011  3.487038385396511E-011
      14 0.1400000E-01  1.96082E-01  8.36122E-05  7.85315E-01  7.85399E-01 volflow Z
         14  Fluid done  1.4000E-02  5.8091E-03
Step     15, t= 1.5000000E-02, DT= 1.0000000E-03, C=  0.024 1.3200E-01 8.8480E-03
             Solving for fluid
         15  Project velx           2.0870E-12   9.5755E-13   2.1795E+00   6   0
         15  Hmholtz velx       1   1.9816E-12   2.7526E-10   1.0000E-08
         15  Project vely           2.2341E-12   9.1877E-13   2.4317E+00   6   0
         15  Hmholtz vely       1   1.8638E-12   2.6079E-10   1.0000E-08
         15  Project velz           7.3997E-03   4.0501E-10   1.8271E+07   6   0
         15  Hmholtz velz       1   7.6036E-10   1.0470E-07   1.0000E-08
         15  U-PRES gmres       1   2.6690E-11   5.1313E-11   1.0000E-08   4.2105E-04   7.5889E-04
          15  DNORM, DIVEX  2.669000890239623E-011  2.668996948897072E-011
      15 0.1500000E-01  1.93713E-01  8.26021E-05  7.85316E-01  7.85399E-01 volflow Z
         15  Fluid done  1.5000E-02  6.1171E-03
Step     16, t= 1.6000000E-02, DT= 1.0000000E-03, C=  0.024 1.4055E-01 8.5521E-03
             Solving for fluid
         16  Project velx           1.4278E-12   7.9313E-13   1.8002E+00   7   0
         16  Hmholtz velx       1   2.6163E-12   2.6755E-10   1.0000E-08
         16  Project vely           1.6042E-12   7.4087E-13   2.1654E+00   7   0
         16  Hmholtz vely       1   2.3466E-12   2.5477E-10   1.0000E-08
         16  Project velz           7.2237E-03   1.3826E-10   5.2246E+07   7   0
         16  Hmholtz velz       1   3.2483E-10   3.9892E-08   1.0000E-08
         16  U-PRES gmres       1   2.1901E-11   5.3006E-11   1.0000E-08   4.1389E-04   7.4697E-04
          16  DNORM, DIVEX  2.190062603618091E-011  2.190051882309305E-011
      16 0.1600000E-01  1.91408E-01  8.16191E-05  7.85317E-01  7.85399E-01 volflow Z
         16  Fluid done  1.6000E-02  6.4211E-03
Step     17, t= 1.7000000E-02, DT= 1.0000000E-03, C=  0.024 1.4961E-01 9.0539E-03
             Solving for fluid
         17  Project velx           1.3666E-12   4.9154E-13   2.7802E+00   8   0
         17  Hmholtz velx       1   1.2733E-12   1.6037E-10   1.0000E-08
         17  Project vely           1.4605E-12   4.9563E-13   2.9467E+00   8   0
         17  Hmholtz vely       1   1.4031E-12   1.6687E-10   1.0000E-08
         17  Project velz           7.0554E-03   1.5474E-10   4.5597E+07   8   0
         17  Hmholtz velz       1   4.5936E-10   5.0088E-08   1.0000E-08
         17  U-PRES gmres       1   1.3609E-11   3.4234E-11   1.0000E-08   4.0889E-04   7.4100E-04
          17  DNORM, DIVEX  1.360943508813963E-011  1.360926198189646E-011
      17 0.1700000E-01  1.89164E-01  8.06623E-05  7.85318E-01  7.85399E-01 volflow Z
         17  Fluid done  1.7000E-02  6.5470E-03
Step     18, t= 1.8000000E-02, DT= 1.0000000E-03, C=  0.024 1.5878E-01 9.1720E-03
             Solving for fluid
         18  Project velx           1.0274E-12   5.3877E-13   1.9069E+00   8   0
         18  Hmholtz velx       1   1.4884E-12   1.8013E-10   1.0000E-08
         18  Project vely           1.0690E-12   5.8445E-13   1.8290E+00   8   0
         18  Hmholtz vely       1   1.8685E-12   2.0648E-10   1.0000E-08
         18  Project velz           6.8946E-03   7.1559E-11   9.6348E+07   8   0
         18  Hmholtz velz       1   1.6445E-10   2.1576E-08   1.0000E-08
         18  U-PRES gmres       1   1.0436E-11   2.4511E-11   1.0000E-08   4.1199E-04   7.6413E-04
          18  DNORM, DIVEX  1.043611579202116E-011  1.043609660969369E-011
      18 0.1800000E-01  1.86979E-01  7.97307E-05  7.85319E-01  7.85399E-01 volflow Z
         18  Fluid done  1.8000E-02  6.5770E-03
Step     19, t= 1.9000000E-02, DT= 1.0000000E-03, C=  0.024 1.6790E-01 9.1209E-03
             Solving for fluid
         19  Project velx           8.7493E-13   3.9461E-13   2.2172E+00   8   0
         19  Hmholtz velx       1   1.0589E-12   1.3349E-10   1.0000E-08
         19  Project vely           8.9401E-13   4.5467E-13   1.9663E+00   8   0
         19  Hmholtz vely       1   1.2473E-12   1.5373E-10   1.0000E-08
         19  Project velz           6.7408E-03   7.4093E-11   9.0978E+07   8   0
         19  Hmholtz velz       1   2.4016E-10   2.3433E-08   1.0000E-08
         19  U-PRES gmres       1   1.0136E-11   2.1459E-11   1.0000E-08   4.1890E-04   7.4482E-04
          19  DNORM, DIVEX  1.013612501741543E-011  1.013595013048983E-011
      19 0.1900000E-01  1.84852E-01  7.88234E-05  7.85320E-01  7.85399E-01 volflow Z
         19  Fluid done  1.9000E-02  6.5510E-03
Step     20, t= 2.0000000E-02, DT= 1.0000000E-03, C=  0.024 1.7691E-01 9.0160E-03
             Solving for fluid
         20  Project velx           5.4623E-13   3.0284E-13   1.8037E+00   8   0
         20  Hmholtz velx       1   9.3158E-13   9.9875E-11   1.0000E-08
         20  Project vely           5.6154E-13   3.4088E-13   1.6473E+00   8   0
         20  Hmholtz vely       1   1.0630E-12   1.1617E-10   1.0000E-08
         20  Project velz           6.5937E-03   2.4260E-10   2.7179E+07   8   0
         20  Hmholtz velz       1   5.4904E-10   7.5425E-08   1.0000E-08
    1 1.00000E-08 1.14059E-11 2.57048E-11 4.43728E-01      20 Divergence
         20  U-PRES gmres       1   1.1406E-11   2.5705E-11   1.0000E-08   4.2105E-04   7.6318E-04
          20  DNORM, DIVEX  1.140603378518951E-011  1.140594271632045E-011
      20 0.2000000E-01  1.82779E-01  7.79396E-05  7.85321E-01  7.85399E-01 volflow Z
         20  Fluid done  2.0000E-02  6.5720E-03

       20  2.0000E-02 Write checkpoint
       FILE:
 hydro0.f00003                                                                  
                                                      

       20  2.0000E-02 done :: Write checkpoint
                              file size =      1.9    MB
                              avg data-throughput =   135.4MB/s
                              io-nodes =    24

Step     21, t= 2.1000000E-02, DT= 1.0000000E-03, C=  0.024 2.0031E-01 2.3399E-02
             Solving for fluid
         21  Project velx           7.6707E-13   3.7846E-13   2.0268E+00   8   0
         21  Hmholtz velx       1   8.7942E-13   1.1567E-10   1.0000E-08
         21  Project vely           7.7903E-13   4.1219E-13   1.8900E+00   8   0
         21  Hmholtz vely       1   9.9778E-13   1.2594E-10   1.0000E-08
         21  Project velz           6.4529E-03   3.3439E-10   1.9298E+07   8   0
         21  Hmholtz velz       1   1.0414E-09   1.0937E-07   1.0000E-08
         21  U-PRES gmres       1   8.8641E-12   1.7381E-11   1.0000E-08   4.2319E-04   7.5603E-04
          21  DNORM, DIVEX  8.864165661558847E-012  8.864093057245260E-012
      21 0.2100000E-01  1.80760E-01  7.70784E-05  7.85321E-01  7.85399E-01 volflow Z
         21  Fluid done  2.1000E-02  6.6080E-03
Step     22, t= 2.2000000E-02, DT= 1.0000000E-03, C=  0.024 2.0941E-01 9.1000E-03
             Solving for fluid
         22  Project velx           1.0020E-12   4.7137E-13   2.1257E+00   8   0
         22  Hmholtz velx       1   1.0187E-12   1.3992E-10   1.0000E-08
         22  Project vely           1.0078E-12   5.0553E-13   1.9935E+00   8   0
         22  Hmholtz vely       1   1.0447E-12   1.4699E-10   1.0000E-08
         22  Project velz           6.3181E-03   4.1853E-10   1.5096E+07   8   0
         22  Hmholtz velz       1   1.6836E-09   1.5292E-07   1.0000E-08
         22  U-PRES gmres       1   1.1124E-11   2.7521E-11   1.0000E-08   4.1294E-04   7.3910E-04
          22  DNORM, DIVEX  1.112350036742130E-011  1.112355704875387E-011
      22 0.2200000E-01  1.78791E-01  7.62391E-05  7.85322E-01  7.85399E-01 volflow Z
         22  Fluid done  2.2000E-02  6.6011E-03
Step     23, t= 2.3000000E-02, DT= 1.0000000E-03, C=  0.024 2.1891E-01 9.4991E-03
             Solving for fluid
         23  Project velx           2.0033E-12   7.0411E-13   2.8452E+00   8   0
         23  Hmholtz velx       1   2.1215E-12   2.2952E-10   1.0000E-08
         23  Project vely           2.0084E-12   6.8921E-13   2.9140E+00   8   0
         23  Hmholtz vely       1   2.1462E-12   2.2652E-10   1.0000E-08
         23  Project velz           6.1890E-03   1.1746E-10   5.2689E+07   8   0
         23  Hmholtz velz       1   5.2150E-10   5.4728E-08   1.0000E-08
         23  U-PRES gmres       1   1.5356E-11   3.7380E-11   1.0000E-08   4.1914E-04   7.4697E-04
          23  DNORM, DIVEX  1.535600085253532E-011  1.535618069214322E-011
      23 0.2300000E-01  1.76872E-01  7.54208E-05  7.85323E-01  7.85399E-01 volflow Z
         23  Fluid done  2.3000E-02  6.6011E-03
Step     24, t= 2.4000000E-02, DT= 1.0000000E-03, C=  0.024 2.2809E-01 9.1808E-03
             Solving for fluid
         24  Project velx           1.3278E-12   9.3083E-13   1.4265E+00   8   0
         24  Hmholtz velx       1   1.9564E-12   2.6374E-10   1.0000E-08
         24  Project vely           1.3393E-12   9.5074E-13   1.4087E+00   8   0
         24  Hmholtz vely       1   2.1517E-12   2.7481E-10   1.0000E-08
         24  Project velz           6.0653E-03   3.8601E-10   1.5713E+07   8   0
         24  Hmholtz velz       1   1.3674E-09   1.3788E-07   1.0000E-08
         24  U-PRES gmres       1   1.9949E-11   4.1854E-11   1.0000E-08   4.1413E-04   7.5293E-04
          24  DNORM, DIVEX  1.994902567500108E-011  1.994889829933690E-011
      24 0.2400000E-01  1.75001E-01  7.46230E-05  7.85324E-01  7.85399E-01 volflow Z
         24  Fluid done  2.4000E-02  6.6140E-03
Step     25, t= 2.5000000E-02, DT= 1.0000000E-03, C=  0.024 2.3713E-01 9.0411E-03
             Solving for fluid
         25  Project velx           1.6757E-12   8.0392E-13   2.0843E+00   8   0
         25  Hmholtz velx       1   2.0914E-12   2.5894E-10   1.0000E-08
         25  Project vely           1.6921E-12   8.2437E-13   2.0525E+00   8   0
         25  Hmholtz vely       1   2.3363E-12   2.7353E-10   1.0000E-08
         25  Project velz           5.9467E-03   5.0424E-11   1.1793E+08   8   0
         25  Hmholtz velz       1   1.3395E-10   1.7128E-08   1.0000E-08
         25  U-PRES gmres       1   1.5574E-11   3.6509E-11   1.0000E-08   4.1199E-04   7.3695E-04
          25  DNORM, DIVEX  1.557425419252328E-011  1.557413483174415E-011
      25 0.2500000E-01  1.73176E-01  7.38447E-05  7.85325E-01  7.85399E-01 volflow Z
         25  Fluid done  2.5000E-02  6.5801E-03
Step     26, t= 2.6000000E-02, DT= 1.0000000E-03, C=  0.024 2.4670E-01 9.5689E-03
             Solving for fluid
         26  Project velx           1.2205E-12   1.0942E-12   1.1154E+00   8   0
         26  Hmholtz velx       1   2.7186E-12   3.3343E-10   1.0000E-08
         26  Project vely           1.2372E-12   1.1241E-12   1.1006E+00   8   0
         26  Hmholtz vely       1   2.7979E-12   3.4316E-10   1.0000E-08
         26  Project velz           5.8330E-03   1.0244E-10   5.6942E+07   8   0
         26  Hmholtz velz       1   3.8475E-10   3.5464E-08   1.0000E-08
         26  U-PRES gmres       1   9.7700E-12   1.9884E-11   1.0000E-08   4.1699E-04   7.7295E-04
          26  DNORM, DIVEX  9.769898352285633E-012  9.770009398970843E-012
      26 0.2600000E-01  1.71396E-01  7.30855E-05  7.85325E-01  7.85399E-01 volflow Z
         26  Fluid done  2.6000E-02  6.6321E-03
Step     27, t= 2.7000000E-02, DT= 1.0000000E-03, C=  0.024 2.5576E-01 9.0561E-03
             Solving for fluid
         27  Project velx           8.7449E-13   6.5973E-13   1.3255E+00   8   0
         27  Hmholtz velx       1   1.5832E-12   2.0449E-10   1.0000E-08
         27  Project vely           8.9548E-13   6.7206E-13   1.3324E+00   8   0
         27  Hmholtz vely       1   1.6389E-12   2.1020E-10   1.0000E-08
         27  Project velz           5.7239E-03   7.8727E-11   7.2705E+07   8   0
         27  Hmholtz velz       1   2.0872E-10   2.4526E-08   1.0000E-08
         27  U-PRES gmres       1   1.2224E-11   2.4136E-11   1.0000E-08   4.2510E-04   7.5197E-04
          27  DNORM, DIVEX  1.222338858502294E-011  1.222351929261660E-011
      27 0.2700000E-01  1.69658E-01  7.23446E-05  7.85326E-01  7.85399E-01 volflow Z
         27  Fluid done  2.7000E-02  6.6199E-03
Step     28, t= 2.8000000E-02, DT= 1.0000000E-03, C=  0.025 2.6478E-01 9.0241E-03
             Solving for fluid
         28  Project velx           6.0816E-13   4.4563E-13   1.3647E+00   8   0
         28  Hmholtz velx       1   1.0679E-12   1.3984E-10   1.0000E-08
         28  Project vely           6.3460E-13   4.9212E-13   1.2895E+00   8   0
         28  Hmholtz vely       1   1.2174E-12   1.6108E-10   1.0000E-08
         28  Project velz           5.6191E-03   2.8628E-10   1.9628E+07   8   0
         28  Hmholtz velz       1   9.1642E-10   1.0748E-07   1.0000E-08
         28  U-PRES gmres       1   1.2698E-11   2.9665E-11   1.0000E-08   4.1604E-04   7.4100E-04
          28  DNORM, DIVEX  1.269758373785236E-011  1.269769385205162E-011
      28 0.2800000E-01  1.67962E-01  7.16215E-05  7.85327E-01  7.85399E-01 volflow Z
         28  Fluid done  2.8000E-02  6.5951E-03
Step     29, t= 2.9000000E-02, DT= 1.0000000E-03, C=  0.025 2.7384E-01 9.0549E-03
             Solving for fluid
         29  Project velx           1.1817E-12   7.7278E-13   1.5291E+00   8   0
         29  Hmholtz velx       1   1.3410E-12   2.0816E-10   1.0000E-08
         29  Project vely           1.2079E-12   7.7686E-13   1.5548E+00   8   0
         29  Hmholtz vely       1   1.3716E-12   2.1027E-10   1.0000E-08
         29  Project velz           5.5185E-03   7.0110E-11   7.8713E+07   8   0
         29  Hmholtz velz       1   2.8204E-10   2.7890E-08   1.0000E-08
         29  U-PRES gmres       1   1.2570E-11   2.9788E-11   1.0000E-08   4.1199E-04   7.3886E-04
          29  DNORM, DIVEX  1.257021803612805E-011  1.257021373763485E-011
      29 0.2900000E-01  1.66307E-01  7.09155E-05  7.85328E-01  7.85399E-01 volflow Z
         29  Fluid done  2.9000E-02  6.6278E-03
Step     30, t= 3.0000000E-02, DT= 1.0000000E-03, C=  0.025 2.8313E-01 9.2869E-03
             Solving for fluid
         30  Project velx           9.4261E-13   6.2417E-13   1.5102E+00   8   0
         30  Hmholtz velx       1   1.2782E-12   1.7280E-10   1.0000E-08
         30  Project vely           9.8026E-13   6.3862E-13   1.5350E+00   8   0
         30  Hmholtz vely       1   1.4106E-12   1.7730E-10   1.0000E-08
         30  Project velz           5.4219E-03   8.6166E-10   6.2924E+06   8   0
         30  Hmholtz velz       1   2.8812E-09   2.9992E-07   1.0000E-08
         30  U-PRES gmres       1   1.1464E-10   2.6503E-10   1.0000E-08   4.4107E-04   7.8297E-04
          30  DNORM, DIVEX  1.146484171228502E-010  1.146395845477926E-010
      30 0.3000000E-01  1.64690E-01  7.02261E-05  7.85328E-01  7.85399E-01 volflow Z
         30  Fluid done  3.0000E-02  6.5820E-03

       30  3.0000E-02 Write checkpoint
       FILE:
 hydro0.f00004                                                                  
                                                      

       30  3.0000E-02 done :: Write checkpoint
                              file size =      1.9    MB
                              avg data-throughput =    79.0MB/s
                              io-nodes =    24

Step     31, t= 3.1000000E-02, DT= 1.0000000E-03, C=  0.025 3.1607E-01 3.2944E-02
             Solving for fluid
         31  Project velx           1.9210E-11   1.9138E-11   1.0037E+00   8   0
         31  Hmholtz velx       1   1.9672E-11   5.0448E-09   1.0000E-08
         31  Project vely           1.7216E-11   1.7152E-11   1.0038E+00   8   0
         31  Hmholtz vely       1   1.8736E-11   4.9216E-09   1.0000E-08
         31  Project velz           5.3290E-03   9.5708E-09   5.5679E+05   8   0
         31  Hmholtz velz       2   3.0379E-10   3.2832E-06   1.0000E-08
         31  U-PRES gmres       3   6.2988E-09   4.6644E-08   1.0000E-08   1.0133E-03   2.0421E-03
          31  DNORM, DIVEX  6.309401183402714E-009  6.298823073726441E-009
      31 0.3100000E-01  1.63111E-01  6.95527E-05  7.85329E-01  7.85399E-01 volflow Z
         31  Fluid done  3.1000E-02  7.9679E-03
Step     32, t= 3.2000000E-02, DT= 1.0000000E-03, C=  0.025 3.2660E-01 1.0526E-02
             Solving for fluid
         32  Project velx           6.6444E-09   2.4759E-09   2.6836E+00   8   0
         32  Hmholtz velx       1   3.9238E-09   6.7918E-07   1.0000E-08
         32  Project vely           6.8730E-09   2.1046E-09   3.2656E+00   8   0
         32  Hmholtz vely       1   3.6166E-09   5.9974E-07   1.0000E-08
         32  Project velz           5.2396E-03   2.5121E-06   2.0858E+03   8   0
         32  Hmholtz velz       3   7.6028E-10   9.3795E-04   1.0000E-08
       32proj_ortho:    2   8 velz   Detect rank deficiency:         NaN  1.0000E+00
       32proj_ortho:    1   8 velz   Detect rank deficiency:         NaN  1.0000E+00
         32  U-PRES gmres      29   9.9376E-09   5.2213E-04   1.0000E-08   8.7926E-03   1.7780E-02
          32  DNORM, DIVEX  2.704668739973178E-005  9.937641774554489E-009
      32 0.3200000E-01  1.61568E-01  6.88948E-05  7.85330E-01  7.85399E-01 volflow Z
         32  Fluid done  3.2000E-02  2.4042E-02
Step     33, t= 3.3000000E-02, DT= 1.0000000E-03, C=  0.025 3.5334E-01 2.6744E-02
             Solving for fluid
         33  Project velx           5.2937E-04   3.4066E-05   1.5540E+01   8   0
         33  Hmholtz velx       3   5.8362E-09   1.0173E-02   1.0000E-08
         33  Project vely           5.4648E-04   3.5738E-05   1.5291E+01   8   0
         33  Hmholtz vely       3   6.1795E-09   1.0217E-02   1.0000E-08
         33  Project velz           5.1642E-03   6.3547E-04   8.1266E+00   6   0
         33  Hmholtz velz       4   1.3787E-09   2.4417E-01   1.0000E-08
       33proj_ortho:    2   7 velz   Detect rank deficiency:         NaN  1.0000E+00
         33  U-PRES gmres     120   3.2992E-04   9.1268E-04   1.0000E-08   3.6037E-02   7.4512E-02
          33  DNORM, DIVEX   2.46689287957593       3.299187314590418E-004
      33 0.3300000E-01  1.59990E-01  6.82220E-05  7.85330E-01  7.85399E-01 volflow Z
         33  Fluid done  3.3000E-02  8.1082E-02
 CFL, Ctarg!   284.332673470552       0.500000000000000     

       34  3.3000E-02 Write checkpoint
       FILE:
 hydro0.f00005                                                                  
                                                      

       34  3.3000E-02 done :: Write checkpoint
                              file size =      1.9    MB
                              avg data-throughput =   147.5MB/s
                              io-nodes =    24

        
        
        
        
        
        
        
        
        
        
        
        
        
        
          11  Emergency exit:          34    time =  3.300000000000002E-002
           9  Emergency exit:          34    time =  3.300000000000002E-002
          14  Emergency exit:          34    time =  3.300000000000002E-002
        
          17  Emergency exit:          34    time =  3.300000000000002E-002
        
        
  Latest solution and data are dumped for post-processing.
          22  Emergency exit:          34    time =  3.300000000000002E-002
  Latest solution and data are dumped for post-processing.
  *** STOP ***
        
  Latest solution and data are dumped for post-processing.
  Latest solution and data are dumped for post-processing.
  Latest solution and data are dumped for post-processing.
  *** STOP ***
          18  Emergency exit:          34    time =  3.300000000000002E-002
          15  Emergency exit:          34    time =  3.300000000000002E-002
  Latest solution and data are dumped for post-processing.
  *** STOP ***
  *** STOP ***
  Latest solution and data are dumped for post-processing.
  *** STOP ***
  *** STOP ***
  *** STOP ***
           5  Emergency exit:          34    time =  3.300000000000002E-002
           8  Emergency exit:          34    time =  3.300000000000002E-002
          10  Emergency exit:          34    time =  3.300000000000002E-002
  Latest solution and data are dumped for post-processing.
  Latest solution and data are dumped for post-processing.
  *** STOP ***
  *** STOP ***
  Latest solution and data are dumped for post-processing.
  *** STOP ***
        
        
        
          13  Emergency exit:          34    time =  3.300000000000002E-002
          23  Emergency exit:          34    time =  3.300000000000002E-002
  Latest solution and data are dumped for post-processing.
  Latest solution and data are dumped for post-processing.
  *** STOP ***
  *** STOP ***
          21  Emergency exit:          34    time =  3.300000000000002E-002
  Latest solution and data are dumped for post-processing.
  *** STOP ***
           0  Emergency exit:          34    time =  3.300000000000002E-002
        
           1  Emergency exit:          34    time =  3.300000000000002E-002
  Latest solution and data are dumped for post-processing.
  *** STOP ***
           3  Emergency exit:          34    time =  3.300000000000002E-002
           7  Emergency exit:          34    time =  3.300000000000002E-002
  Latest solution and data are dumped for post-processing.
           6  Emergency exit:          34    time =  3.300000000000002E-002
  *** STOP ***
  Latest solution and data are dumped for post-processing.
  Latest solution and data are dumped for post-processing.
           2  Emergency exit:          34    time =  3.300000000000002E-002
  *** STOP ***
  *** STOP ***
  Latest solution and data are dumped for post-processing.
           4  Emergency exit:          34    time =  3.300000000000002E-002
  *** STOP ***
  Latest solution and data are dumped for post-processing.
  *** STOP ***
        
          20  Emergency exit:          34    time =  3.300000000000002E-002
          12  Emergency exit:          34    time =  3.300000000000002E-002
  Latest solution and data are dumped for post-processing.
  *** STOP ***
  Latest solution and data are dumped for post-processing.
  *** STOP ***
        
          19  Emergency exit:          34    time =  3.300000000000002E-002
  Latest solution and data are dumped for post-processing.
          16  Emergency exit:          34    time =  3.300000000000002E-002
  *** STOP ***
  Latest solution and data are dumped for post-processing.
  *** STOP ***
  Latest solution and data are dumped for post-processing.
  *** STOP ***
runtime statistics:
 total time  0.353340010507202     
 mxmf time           0  0.000000000000000E+000  0.000000000000000E+000
 tgop time        5575  3.266096115112305E-002  9.243493569901651E-002
 inv3 time         273  1.613378524780273E-003  4.566079347946892E-003
 invc time          99  9.999275207519531E-004  2.829930070236333E-003
 mltd time         918  2.628493309020996E-002  7.438991427118383E-002
 cdtp time         918  2.229833602905273E-002  6.310730561490778E-002
 eslv time          36  0.120205163955688       0.340196865288875     
 pres time          33  0.150884628295898       0.427023897121956     
 crsl time         185  5.387544631958008E-003  1.524747968458045E-002
 crsl min   5.160808563232422E-003
 crsl max   5.882263183593750E-003
 crsl avg   5.474865436553955E-003
 hmhz time          33  5.243468284606934E-002  0.148397241429868     
 spro time          34  4.618167877197266E-004  1.307003945171144E-003
 usbc time          33  8.642435073852539E-003  2.445925968431016E-002
 usbc min   7.734060287475586E-003
 usbc max   9.352207183837891E-003
 usb  avg   8.474369843800863E-003
 axhm time         694  3.651022911071777E-002  0.103328884431483     
 advc time          99  1.854228973388672E-002  5.247718679599906E-002
 vdss time         274  2.405118942260742E-002  6.806811769797348E-002
 vdss min   1.815438270568848E-002
 vdss max   2.539277076721191E-002
 vdss avg   2.211845914522807E-002
 dsum time        1141  4.439687728881836E-002  0.125649165021218     
 dsum min   3.167104721069336E-002
 dsum max   4.561710357666016E-002
 dsum avg   4.384730259577433E-002
 dadd time           0  1.598310470581055E-002  4.523434717417818E-002
 ddsl time         185  5.954694747924805E-002  0.168525911893678     
 solv time         185  6.815433502197266E-003  1.928859823265994E-002
 prep time          35  6.918621063232422E-002  0.195806329809666     

MPI timings
 total comm time  4.600620269775391E-002  0.166948377715475     
 comm min   3.311204910278320E-002
 comm max   5.898952484130859E-002
 comm avg   4.842885335286459E-002
 barrier time           0  0.000000000000000E+000  0.000000000000000E+000
 barrier min   0.000000000000000E+000
 barrier max   0.000000000000000E+000
 barrier avg   0.000000000000000E+000
 waitall time       11195  3.656482696533203E-002  0.794780373541179     
 waitall min   2.262520790100098E-002
 waitall max   4.934859275817871E-002
 waitall avg   3.865465521812439E-002
 allreduce  time           0  0.000000000000000E+000  0.000000000000000E+000
 allreduce  min   0.000000000000000E+000
 allreduce  max   0.000000000000000E+000
 allreduce  avg   0.000000000000000E+000
 allreduce_sync  time  0.000000000000000E+000  0.000000000000000E+000
 allreduce_sync  min   0.000000000000000E+000
 allreduce_sync  max   0.000000000000000E+000
 allreduce_sync  avg   0.000000000000000E+000

#  nid       tusbc       tdadd       tcrsl       tvdss       tdsum        tgop    qqq    F
           0  8.6424E-03  1.5983E-02  5.3875E-03  2.4051E-02  4.4397E-02  3.2661E-02 qqq
           1  8.8966E-03  1.6063E-02  5.1608E-03  2.5393E-02  4.4976E-02  5.2797E-02 qqq
           2  8.3587E-03  1.6842E-02  5.6503E-03  2.0835E-02  4.4769E-02  5.7136E-02 qqq
           3  9.3522E-03  1.6212E-02  5.3024E-03  2.1300E-02  3.1671E-02  3.5687E-02 qqq
           4  9.0003E-03  1.6313E-02  5.3015E-03  2.4041E-02  4.4778E-02  5.5589E-02 qqq
           5  8.2464E-03  1.6971E-02  5.8258E-03  1.9455E-02  4.4560E-02  5.9780E-02 qqq
           6  8.6889E-03  1.6192E-02  5.5237E-03  2.4356E-02  4.5617E-02  5.5960E-02 qqq
           7  8.7538E-03  1.5855E-02  5.2750E-03  2.5064E-02  4.5087E-02  5.4449E-02 qqq
           8  8.2150E-03  1.6556E-02  5.7075E-03  2.0328E-02  4.4294E-02  5.8549E-02 qqq
           9  8.5251E-03  1.6092E-02  5.3415E-03  2.2626E-02  4.5111E-02  5.6242E-02 qqq
          10  8.5392E-03  1.6140E-02  5.5597E-03  2.3234E-02  4.4845E-02  5.5619E-02 qqq
          11  7.9479E-03  1.6839E-02  5.8823E-03  1.8791E-02  4.4739E-02  6.0194E-02 qqq
          12  8.7032E-03  1.5378E-02  5.2822E-03  2.3747E-02  4.4782E-02  4.9581E-02 qqq
          13  8.7762E-03  1.5500E-02  5.2562E-03  2.4471E-02  4.4170E-02  4.9750E-02 qqq
          14  8.1465E-03  1.5933E-02  5.6129E-03  1.9685E-02  4.3852E-02  5.3759E-02 qqq
          15  8.3561E-03  1.5804E-02  5.4088E-03  2.1786E-02  4.4206E-02  5.3200E-02 qqq
          16  8.5623E-03  1.5433E-02  5.2996E-03  2.3059E-02  4.3695E-02  5.1350E-02 qqq
          17  7.8588E-03  1.6481E-02  5.7223E-03  1.8388E-02  4.4183E-02  5.7596E-02 qqq
          18  8.5285E-03  1.5386E-02  5.4266E-03  2.3645E-02  4.4376E-02  5.1830E-02 qqq
          19  8.6548E-03  1.5412E-02  5.3701E-03  2.4292E-02  4.3998E-02  5.1528E-02 qqq
          20  8.0819E-03  1.6098E-02  5.5826E-03  1.9737E-02  4.3289E-02  5.5220E-02 qqq
          21  8.3082E-03  1.5827E-02  5.4238E-03  2.1808E-02  4.4188E-02  5.3329E-02 qqq
          22  8.5077E-03  1.5384E-02  5.3835E-03  2.2597E-02  4.3298E-02  5.2802E-02 qqq
          23  7.7341E-03  1.6088E-02  5.7104E-03  1.8154E-02  4.3452E-02  5.6648E-02 qqq
  
call exitt: dying ...
  
backtrace(): obtained 10 stack frames.
[0x20255fc9]
[0x2038cba2]
[0x201380ce]
[0x2011a53a]
[0x20003ddf]
[0x20000e1e]
[0x20000c5b]
[0x20000aae]
[0x207b7d11]
[0x20000989]
  
total elapsed time             :   5.61988E-01 sec
total solver time incl. I/O    :   3.53340E-01 sec
time/timestep                  :   1.03924E-02 sec
avg throughput per timestep    :   2.01238E+05 gridpts/CPUs
max resident memory            :   1.03506E+01 MB

Application 7145931 resources: utime ~20s, stime ~11s, Rss ~10672, inblocks ~69765, outblocks ~51109


More information about the Nek5000-users mailing list