From nek5000-users at lists.mcs.anl.gov Wed Jul 1 08:10:48 2015 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 1 Jul 2015 15:10:48 +0200 (CEST) Subject: [Nek5000-users] how to use nek only with fem Message-ID: Hi Nek's I wanted to know if there is a way to use nek only with finite elements, without spectral. If yes, which is the best way to do it? -- Mr. Mirko FaranoPh.D. student at Politecnico di BariTel.: +39-0883529289 / +39-0805963462Mob.: +39-3202342719Department of Mechanics Mathematics and Management - Fluid Machinery and Energy SystemsVia Re David, 200 - 70125 BARI (Italy)mirko.farano at poliba.itIn joint supervision with theEcole Nationale Sup?rieure d'Arts et M?tiers de Paris - ParistechTel.: +33-144246436Mob.: +33-789822134Dynfluid Laboratory151 Boulevard de l'H?pital - 75013 Paris (France)mirko.farano at ensam.eu -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Wed Jul 1 10:55:35 2015 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 1 Jul 2015 15:55:35 +0000 Subject: [Nek5000-users] Spectral interpolation of velocity gradients In-Reply-To: References: Message-ID: Hi Philipp, Your approach sounds correct. Nek automatically migrates particles to wherever the geometry is. This is done using a relatively efficient generalized all-to-all (that requires only log P message exchanges). Point identification is done through a combination of a hash table, to identify candidate elements, and a very robust Newton iteration to identify the computational point in r,s,t, space that corresponds to x,y,z. Of these processes, findpts is the most expensive. Subsequent interpolation onto multiple fields is relatively cheap. I have some code developed by another user that will interpolate many fields in a single go --- there are computational advantages for doing so --- but it is not yet general purpose (by the author's own admission). My hope is to get this into the repo soon but I could provide it off-list if you are keen to dig into some C code -- only if you are keen. Otherwise, I think there are other optimizations that could be tackled, and we will indeed be working on these over the next few weeks. Paul ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Monday, June 29, 2015 12:58 PM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] Spectral interpolation of velocity gradients Hi Neks, I study particles in a turbulent flow and need to know velocity gradients dvi_dxj at particle positions x_p. In order to do this, I modified the subroutine interp_v(uvw,xyz,n) from the particle tracking example ''hemi''. First, I calculate the velocity gradients with the subroutine gradm1(): call gradm1(vxx,vxy,vxz,vx) call gradm1(vyx,vzy,vzz,vy) call gradm1(vzx,vzy,vzz,vz) Second, I interpolate the velocity gradients to the particle positions with the subroutine intpts(): call opcopy(wrk(1,1),wrk(1,2),wrk(1,3),vxx,vyx,vzx) call intpts(wrk,ndm,pts,npart,outint,ifjac,ifpts,ihandle) I have a few questions regarding this approach: 1. Is intpts() capable of interpolating derived fields like the velocity gradient field? 2. Do you have a good explanatory reference where the mathematical principles behind intpts() are explained? 3. As far as I understand, the fields vx, vy, vz and consequently the derived fields vxx, vxy, vxz, ... are local and defined on the local coordinate fields xm1, ym1, zm1. What is happens if an interpolation point is on a different processor and outside the coordinates stored in xm1, ym1, zm1? Best regards, Philipp Weiss -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Tue Jul 7 09:52:37 2015 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 7 Jul 2015 16:52:37 +0200 Subject: [Nek5000-users] Large simulations Message-ID: Dear All, Is it normal for a nek simulation (220 000 elements with lx1=12) to have long initial setup times (basically gs_setup) such as 5 hours when running on 32768 cores, even when restarting? As expected, more elements and/or more cores used results in longer setup times. My problem is that I wish to carry out simulations with a higher number of elements but I am limited to 24 hour runs and must carry out restarts. Is there a way to carry out faster restarts? Thank you, jp -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Tue Jul 7 11:05:13 2015 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 7 Jul 2015 16:05:13 +0000 Subject: [Nek5000-users] Large simulations In-Reply-To: References: Message-ID: Hi JP, That doesn't sound normal at all... 200K elements is not that large. I'm suspecting that the time is spent in factoring the XXt-based coarse grid operator..., but that should really take only 20 minutes for the numbers you report (which is still too long). My guess is that a switch to AMG for the coarse-grid solve would fix your problem --- My first guess about the long time is that somehow the partitioning is messed up --- XXt depends crucially on having a good partition and genmap is designed to provide this. Please let me know if you have any issues switching to AMG (which we typically only do at around a million elements or so...) Best, Paul ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Tuesday, July 07, 2015 9:52 AM To: nek5000-users Subject: [Nek5000-users] Large simulations Dear All, Is it normal for a nek simulation (220 000 elements with lx1=12) to have long initial setup times (basically gs_setup) such as 5 hours when running on 32768 cores, even when restarting? As expected, more elements and/or more cores used results in longer setup times. My problem is that I wish to carry out simulations with a higher number of elements but I am limited to 24 hour runs and must carry out restarts. Is there a way to carry out faster restarts? Thank you, jp -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Thu Jul 9 13:53:38 2015 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 9 Jul 2015 18:53:38 +0000 Subject: [Nek5000-users] gradm1 vs opdiv for computing velocity divergence Message-ID: Dear all, I am new to Nek5000 and am trying to output the velocity divergence of my 2D, unsteady N-S solution for a flat plate with elliptic leading edge using PN-PN-2 formulation. I compute divergence using gradm1() and opdiv() and the values I get from the two don't seem to match the DIVERGENCE and HELMHOLTZ (p021 and p022) residuals in the .rea file (which are 1e-9 and 1e-11). I get div ~ 1 using gradm1() and div~1e-2 using opdiv() and the divergence is close to the elliptic leading edge. Am I missing something(scaling factor maybe?) in the computation of divergence here? subroutine userchk include 'SIZE' include 'TOTAL' common /scren/ vort (lx1,ly1,lz1,lelv,3) ! x y z components $ , work1(lx1,ly1,lz1,lelv) $ , work2(lx1,ly1,lz1,lelv) $ , dudx(lx1,ly1,lz1,lelv) $ , dvdy(lx1,ly1,lz1,lelv) $ , div1(lx1,ly1,lz1,lelv) $ , div2(lx2,ly2,lz2,lelv) $ , div3(lx1,ly1,lz1,lelv) n = nx1*ny1*nz1*nelt ntot2 = nx2*ny2*nz2*nelt c Compute divergence using gradm1() call gradm1(dudx,work1,work2,vx) call gradm1(work1,dvdy,work2,vy) div1 = dudx + dvdy c Compute divergence using opdiv() call opdiv(div2,vx,vy,vz) c opdiv returns bm2*div call col2(div2,bm2inv,ntot2) call mappr(div3,div2,work1,work2) if(nid .eq.0)write(6,*)'Done mapping div2 to xm1' call outpost(div1,div3,vx,pr,vort(:,:,:,:,1), & 'divvort') return end This is from the output file showing the DNORM, DIVEX Step 2, t= 4.0340639E+00, DT= 1.0892282E-04, C= 0.500 2.1168E+01 2.1168E+01 Solving for fluid F T T 9.999999999999999E-012 p22 2 1 2 Hmholtz VELX: 42 6.4400E-12 8.2479E-02 1.0000E-11 9.999999999999999E-012 p22 2 1 2 Hmholtz VELY: 40 6.3878E-12 9.2957E-02 1.0000E-11 2 U-PRES gmres: 120 6.0275E-08 1.0000E-09 6.4717E-04 5.7658E+00 1.6938E+01 2 DNORM, DIVEX 6.027476742433520E-008 6.027476742402435E-008 2 4.0341E+00 2.0912E+01 Fluid done Thanks Prahladh Iyer Postdoctoral Researcher National Institute of Aerospace, Hampton, Virginia Resident: NASA Langley Research Center Email : prahladh.s.iyer at nasa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Thu Jul 23 09:03:44 2015 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 23 Jul 2015 16:03:44 +0200 Subject: [Nek5000-users] Dynamo set up with weird results Message-ID: Hello everyone, I am currently looking into using Nek5000 for a (kinematic) dynamo problem in a spheroid. I started by combining the mhd example from the examples with a working case which only solved for the velocity the following way: - lbelv=lelv, lbx1=lx1 etc. in SIZE - adding (dummy) bc's for Temperature to the rea file and setting ifheat=.false. in usrchk - adding bc's for the magnetic field to the rea file as Passive Scalar 1, after the temperature. The string is 'E ' inside the Volume and 'v ' at the boundary - ux,uy,uz=0.0 in userbc for both (ifield .eq. 1) and (ifield .eq. ifldmhd) - ux,uy,uz=0.0 in useric for both (ifield .eq. 1) and (ifield .eq. ifldmhd) - param(29) is set to the magn. viscosity My actual problem is of course more complicated in regards to init. and boundary conditions, but this simple example produces weird output. The magnetic field should stay zero, but grows with the same rate as the velocityfield (computed via glsc3(bm1,bx,bx,n)+glsc3(bm1,by,by,n)+glsc3(bm1,bz,bz,n) and (glsc3(bm1,vx,vx,n)+glsc3(bm1,vy,vy,n)+glsc3(bm1,vz,vz,n))). I then removed the custom forcing to the velocity in userf, which caused the simulation to produce NaNs after 4 timesteps (everything is zero before, as expected). Without the additions for the magnetic field, the case runs fine with the same resolution and timestep length (but has a slightly lower growthrate for the velocity). Has anyone seen similar behaviour? I think that I am missing something obvious in the the mhd setup. Cheers, Jan From nek5000-users at lists.mcs.anl.gov Sun Jul 26 15:25:52 2015 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sun, 26 Jul 2015 20:25:52 +0000 (UTC) Subject: [Nek5000-users] simplify viscosity terms Message-ID: Hi,I want to reduce navier stokes to something like boundary layer equations.I want to keep all the term (3D), just reduce viscosity term to something line mu*d2u/dy2 (omitting? other derivatives). Would you help me where to start? Hamid -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Sun Jul 26 23:08:48 2015 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 27 Jul 2015 00:08:48 -0400 Subject: [Nek5000-users] userf Questions Message-ID: Hi All, Hate to be so long winded, but I'm seeing problems with my code and was wondering if anyone could help... I'm modeling a wind turbine using a modified actuator line method, where I describe the blades (3) as a set of rotating rectangular outlines, advanced in space at each time step. I'm selecting GLL points which fall within my outlines, reading in the velocities, calculating the body forces, then assigning them back to ffx,ffy,ffz. This generates a downstream flowfield and wake in NEK... Looking at rough results, they compare (velocity and vorticity plots and isosurfaces) favorably with other work. However, during some recent debugging to add some features to my code, I noticed that when I repeatedly ran short runs and printed results at the same time step, my point selections for the 3 blades varied with each run, even though I made no other changes to the code or any runtime parameters. A chunk of my point selection for a given area of blade passage looks like: ---------------------F77 Code In userf----------------------------------------------------------------------------------- C Test for location in Quadrant... C if (thetacalc .GE. (pi/2.0 - epsim2)) then C C Close to Vertical Axis... C C Sin+ ... C xlo = centerx + (rhubm*SIN(thetacalc)) xhi = centerx + (bladeradius*SIN(thetacalc)) ymax=centery+(1./SIN(thetacalc))*(chordbig/2.0) ymin=centery-(1./SIN(thetacalc))*(chordbig/2.0) if (((x.GE.xlo).AND.(x.LE.xhi)).AND. & ((y.GE.ymin).AND.(y.LE.ymax))) then C C Calculate forces... C radm = SQRT((x-centerx)**2.0 + (y-centery)**2.0) vthetam = SQRT(ux**2.0 + uy**2.0) psim = ATAN(uz/(avelm*radm + vthetam)) .... .......... .......... -----------------------F77 Code Continues------------------------------------------------------ The xlo,xhi,ymax,ymin values are calculated based on the rotational location of the blade, which changes at each time step, and defines an outline of the blade. Then the IF statements test the x,y point (z-axis location has already been tested) to be within this blade outline, and when a point "HITS" any one of these outlines for the 3 blades, it calculates the forces, prints a diagnostic line, etc... A typical print clip, which only runs at one time step I define and prints for the "Hit" point, is: ---------------------F77 Code In userf----------------------------------------------------------------------------------- C Print output C if(istep.eq.pstepm)then WRITE(4,1000)'Q1V',x,y,z,ux,uy,uz,psim,ffx,ffy,ffz endif ........ ........ -----------------------F77 Code Continues------------------------------------------------------ This tells me the quadrant and general area of the point; because of geometric issues, I have separate selection (and associated print) routines for each Cartesian quadrant which are all tested in turn... So... My problem is if I run this code over and over, and print at say istep=10, the code shows a different number of points selected for each of the 3 blades, on each run! Oddly, there are repeated patterns, and the total is always the same; for this particular setup at this istep, it was 204 points for all 3 blades. An example of the differences is: Blade 1 Blade 2 Blade 3 ----------------------------------------------------------------------------- Run 1 76 125 3 Run 2 32 110 62 Run 3 78 79 47 I've tried a bunch of things to fix it, including going back to a previous version which had no SAVE statements in it (had more manual parameter input); compiling between each run, etc... All my code is within userf. I'm running this in MPI mode on a large SMP server/workstation; compiled with gfortran. I've run this out to 10000 cycles with repeated runs and have always gotten qualitatively identical results. I ran into this problem when I was debugging a new version which added a bunch of automated features I've been working on. But, when going back to previous versions and performing repeated short runs, I discovered that it did it there too... Am I missing something here with MPI and my selection code not working correctly because of the way userf is called? My understanding of the code is that at each time step, all the GLL points and elements are cycled through with a call to userf for each point; the forces are brought back in as ffx,ffy,ffz. If I look qualitatively at a typical run through time, you can see each blade rotating step-by-step, and see the physical outline and edges of the blades. It makes me think that my selection code (which I've debugged and checked out a lot in the past) is basically correct; just don't know why it would select different points with those IF statements on repeated runs. Now, I can see some roundoff-resolution issues where it might vary one or two points between runs, but here the differences are huge. Thanks for listening ;-) Murph Murphy Leo O'Dea PhD Student Mechanical Engineering Oakland University mlodea at oakland.edu From nek5000-users at lists.mcs.anl.gov Tue Jul 28 09:49:51 2015 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 28 Jul 2015 16:49:51 +0200 Subject: [Nek5000-users] Dynamo set up with weird results In-Reply-To: References: Message-ID: Hi again, I was able to solve the problem: I wasn't aware that the forcing in userf is also added to the induction equation. To avoid this, one has to use: ffx=0.0d0 ffy=0.0d0 ffz=0.0d0 if (ifield .eq. 1) then c----velocity forcing here ffx=... ffy=... ffz=... endif as in the mhd example case. My example seems to be running fine (for now). Cheers, Jan From nek5000-users at lists.mcs.anl.gov Thu Jul 30 20:15:47 2015 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 30 Jul 2015 21:15:47 -0400 Subject: [Nek5000-users] Conjugate Heat Transfer Example: **ERROR** Failed in HMHOLTZ: TEMP Message-ID: Hello Nek5K Users, I am having trouble running the "conj_ht" example. Attached is a copy of the output. Without making any changes (other than number of timesteps) to the .rea file I seem to be getting errors when I seek to compute the Temperature field. As the subject states - it looks like there is an error in the HMHOLTZ step when performing the "cggo" subroutine. Any help you can provide would be much appreciated. Best, Saumil -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: conj_ht.output Type: application/octet-stream Size: 133160 bytes Desc: not available URL: