[Nek5000-users] mesh problems
nek5000-users at lists.mcs.anl.gov
nek5000-users at lists.mcs.anl.gov
Thu Dec 27 05:57:06 CST 2012
Hi Joerg,
We are presently working on all aspects of the workflow for large
element counts. (Yours is > 2.5x larger than any previous ones
that I'm aware of.)
You can test your prex theory by running a case with just
a few slabs in z.
Typically I would have n2to3 build the mesh with fairly
large spacing in z and then use usrdat2() to recompress
to the desired z range --- the reason for this is that
genmap needs to identify the mesh topology based on the
mesh geometry (i.e., the vertex locations), which requires
distinguishing "coincident" vertices by the Euclidean
distance separating them. There are tolerances (as prompted
in genmap) that give some control in making the proper
distinction and the tolerance are relative, based on local
edge length of the given elements. In any case, this is a
possible source of the error you're encountering. Increasing
the z spacing reduces the dimensionality of the difficulty
so that z-separation is obvious.
Regarding the long runtime for matlab - yes, this is possible.
(Obviously, extending this phase to parallel execution is a
high priority for us.) It's also possible that if the graph
of the mesh were messed up (e.g., from genmap) then the matlab
phase might take longer to find optimal coarsening/interpolation
for amg.
A reasonable stategy at this point might be to increase the
polynomial order, which would allow, say, a factor (10/8)^3
or (12/10)^3 reduction in the number of elements you need and
thus significantly cut down on some of the difficulties you're
encountering.
hth,
Paul
On Thu, 27 Dec 2012, nek5000-users at lists.mcs.anl.gov wrote:
> Dear Neks,
>
> I generated a big mesh with 7948800 elements for a cylindrical cell of aspect ratio 3, i.e.
>
> x between -1.5 and 1.5
> y between -1.5 and 1.5
> z between 0 and 1
>
> prex, n2to3, reatore2, amgdump and genmap worked. The AMG preconditioning with go.m worked as well (runtime 3.5 weeks).
>
> prex started out of 3 basic elements which were refined with multi split into
>
> Element1: 90, 90 subelements
> Element2: 70, 90
> Element3: 70, 90
> stretch, r, factor 3
>
> I started the job on BG/Q and get the following error messages in the output file:
>
> bash-4.1$ m G3_96.out.0
> /----------------------------------------------------------\
> | _ __ ______ __ __ ______ ____ ____ ____ |
> | / | / // ____// //_/ / ____/ / __ \ / __ \ / __ \ |
> | / |/ // __/ / ,< /___ \ / / / // / / // / / / |
> | / /| // /___ / /| | ____/ / / /_/ // /_/ // /_/ / |
> | /_/ |_//_____//_/ |_|/_____/ \____/ \____/ \____/ |
> | |
> |----------------------------------------------------------|
> | |
> | NEK5000: Open Source Spectral Element Solver |
> | COPYRIGHT (c) 2008-2010 UCHICAGO ARGONNE, LLC |
> | Version: 1.0rc1 / SVN r838 |
> | Web: http://nek5000.mcs.anl.gov |
> | |
> \----------------------------------------------------------/
>
>
> Number of processors: 65536
> REAL wdsize : 8
> INTEGER wdsize : 4
>
>
> Beginning session:
> /homec/hil02/hil020/nek5_svn/g396_r097_Lx1_08_Ra1e9/G3_96.rea
>
>
> timer accuracy: 1.4962500E-07 sec
>
> read .rea file
> read .re2 file
> byte swap: T -0.2931277218E+36 6.543210030
> nelgt/nelgv/lelt: 7948800 7948800 150
> lx1 /lx2 /lx3 : 4 2 4
>
> mapping elements to processors
> element load imbalance: 1 121 122
> done :: mapping elements to processors
>
> reading mesh
> reading curved sides
> reading bc for ifld 1
> reading bc for ifld 2
> done :: read .re2 file
>
> 0 objects found
> done :: read .rea file 496.34 sec
>
> Reset the target Courant number to .5
> setup mesh topology
> Right-handed check complete for 7948800 elements. OK.
> setvert3d: 4 152084809 215675209 152084809 152084809
> call usrsetvert
> done :: usrsetvert
>
> gs_setup: 50336837 unique labels shared
> pairwise times (avg, min, max): 0.00330717 0.000218656 0.00853093
> crystal router : 0.0077495 0.00761262 0.00818637
> used all_to_all method: crystal router
> setupds time 6.1645E-01 seconds 0 4 152084809 7948800
> 8 max multiplicity
> done :: setup mesh topology
>
> call usrdat
> done :: usrdat
>
> generate geomerty data
> vol_t,vol_v: 7.06857958483581861 7.06857958483581861
> done :: generate geomerty data
>
> call usrdat2
> done :: usrdat2
>
> regenerate geomerty data 1
> vol_t,vol_v: 7.06857958483581861 7.06857958483581861
> done :: regenerate geomerty data 1
>
> verify mesh topology
> -1.50001142304138035 1.50001142290878886 Xrange
> -1.50001142298026835 1.50001142292693612 Yrange
> 0.000000000000000000E+00 1.00000000000000000 Zrange
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> i,j,k,ie: 4 2 1 954634
> i,j,k,ie: 1 2 1 916744
> i,j,k,ie: 2 4 2 972970
> i,j,k,ie: 2 1 1 931570
> WARNING1 Element mesh mismatch at:
> i,j,k,ie: 4 2 1 957874
> WARNING1 Element mesh mismatch at:
> i,j,k,ie: 4 2 1 996664
> Near X = -.12308168E-01 -.34092836 .60036000E-01, d: .12947845E-04 .00000000E+00 81.225474
> WARNING1 Element mesh mismatch at:
> Near X = .11336871E-01 -.82826699 .61838637E-01, d: .00000000E+00 -.11766071E-04 88.069931
> Near X = .97128875E-05 .82845591 .60036000E-01, d: .00000000E+00 -.20015559E-04 88.069931
> i,j,k,ie: 2 1 1 981007
> WARNING1 Element mesh mismatch at:
> i,j,k,ie: 2 1 1 939607
> i,j,k,ie: 4 2 4 995854
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> i,j,k,ie: 2 4 3 972989
> WARNING1 Element mesh mismatch at:
> i,j,k,ie: 2 4 4 931590
> Near X = .12193251E-01 .42841310 .66558003E-01, d: .00000000E+00 -.12947474E-04 81.991313
> WARNING1 Element mesh mismatch at:
> i,j,k,ie: 4 2 4 916204
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> i,j,k,ie: 1 2 4 958144
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> Near X = .12320624E-01 .31593135 .73282003E-01, d: .12573222E-04 .00000000E+00 81.079569
> i,j,k,ie: 3 2 4 541894
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> i,j,k,ie: 2 1 1 981003
> WARNING1 Element mesh mismatch at:
> i,j,k,ie: 2 4 1 939603
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> i,j,k,ie: 2 4 4 890151
> WARNING1 Element mesh mismatch at:
> Near X = -.62894639E-05 -1.1769232 .60036000E-01, d: .15284684E-04 .00000000E+00 92.459135
> WARNING1 Element mesh mismatch at:
> Near X = .63923475E-05 1.1769232 .60036000E-01, d: .00000000E+00 -.15284684E-04 92.458255
> i,j,k,ie: 3 2 4 2524
> i,j,k,ie: 3 2 4 624964
> Near X = -.10302672E-04 .82845591 .60036000E-01, d: .20015559E-04 .00000000E+00 88.062609
> i,j,k,ie: 2 4 4 62171
> i,j,k,ie: 2 4 4 600363
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> WARNING1 Element mesh mismatch at:
> ---------------------------------------------------------
> and so on…….
>
>
> The job hangs at this position and does not continue. Can it be that the precision of the coordinate ranges is not sufficient?
> For a job that runs well (here an aspect ratio one cell as an example) I got in the output file the following coordinates.
>
> ……
>
> verify mesh topology
> -0.500000000000000000 0.500000000000000000 Xrange
> -0.500000000000000000 0.500000000000000000 Yrange
> 0.000000000000000000E+00 1.00000000000000000 Zrange
> done :: verify mesh topology
>
> ……
>
> i.e. precise bounds down to the last digit.
> Note also that I had to adapt the programs n2to3 and reatore2, e.g.:
>
> [josc1234 at reynolds n2to3]$ diff n2to3.f n2to3.f_orig
> 31c31
> < parameter(nelm=99999)
> ---
>> parameter(nelm=9999)
>
> [josc1234 at reynolds reatore2]$ diff reatore2.f reatore2.f_orig
> 19c19
> < parameter(lelt=10000000)
> ---
>> parameter(lelt=1000000)
> 118c118
>
>
> I wonder if you have an idea where things went wrong? Is it already the prex?
> Thanks for your help in advance.
>
> Best regards, Joerg.
>
>
> _______________________________________________
> Nek5000-users mailing list
> Nek5000-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>
More information about the Nek5000-users
mailing list