From nek5000-users at lists.mcs.anl.gov Sun Nov 2 12:11:32 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sun, 2 Nov 2014 12:11:32 -0600 Subject: [Nek5000-users] Submitting Job Nek Enabled Moab In-Reply-To: References: Message-ID: Friedrich, Sorry about the late reply. I looked at the config logs and don't see anything wrong with it. I can't see any architecture info since the warwick site needs authentication. It might be worthwhile if you can test with static builds in case the shared ones are still failing. I am not entirely sure why you get memory errors (in one of your previous threads). Vijay On Thu, Oct 30, 2014 at 7:44 AM, wrote: > Hi Vijay, > > > Firstly thank you for your response! > > > I have attempted all you suggested and this still does not seem to fix my > issues. > > > When I run ldd nek5000 I see: > > > linux-vdso.so.1 => (0x00007fff635ff000) > libz.so.1 => /home/eng/esumgy/pre-moab/zlib/zlib-1.2.8/libz.so.1 > (0x00007fdd5a627000) > libm.so.6 => /lib64/libm.so.6 (0x00007fdd5a37a000) > libsz.so.2 => > /home/eng/esumgy/pre-moab/szip/szip-2.1/gcc/lib/libsz.so.2 > (0x00007fdd5a166000) > libgpfs.so => /usr/lib64/libgpfs.so (0x00007fdd5a05a000) > libdl.so.2 => /lib64/libdl.so.2 (0x00007fdd59e55000) > libmpi_f77.so.0 => > /hpcwarwick/ompi/1.4.5/intel/11.1/lib/libmpi_f77.so.0 (0x00007fdd59c19000) > libmpi.so.0 => /hpcwarwick/ompi/1.4.5/intel/11.1/lib/libmpi.so.0 > (0x00007fdd59945000) > libopen-rte.so.0 => > /hpcwarwick/ompi/1.4.5/intel/11.1/lib/libopen-rte.so.0 (0x00007fdd596dc000) > libopen-pal.so.0 => > /hpcwarwick/ompi/1.4.5/intel/11.1/lib/libopen-pal.so.0 (0x00007fdd5945e000) > libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fdd59246000) > libutil.so.1 => /lib64/libutil.so.1 (0x00007fdd59042000) > libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fdd58d3c000) > libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdd58b26000) > libintlc.so.5 => > /gpfs/hpcwarwick/intel/Compiler/11.1/059/lib/intel64/libintlc.so.5 > (0x00007fdd589e8000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdd587cb000) > libc.so.6 => /lib64/libc.so.6 (0x00007fdd58452000) > librt.so.1 => /lib64/librt.so.1 (0x00007fdd58248000) > /lib64/ld-linux-x86-64.so.2 (0x00007fdd5a83f000) > libimf.so => > /gpfs/hpcwarwick/intel/Compiler/11.1/059/lib/intel64/libimf.so > (0x00007fdd57eb5000) > libsvml.so => > /gpfs/hpcwarwick/intel/Compiler/11.1/059/lib/intel64/libsvml.so > (0x00007fdd57c9d000) > > > > The compiler being used is ompi/1.4.5/intel/11.1 > > > See also the moab and hdf5 config.log attached! > > > System information can be found - > http://www2.warwick.ac.uk/fac/sci/csc/local/hpc > > > Thank you for your help I am happy maybe to finally solve this issue > > > Friedrich > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > From nek5000-users at lists.mcs.anl.gov Mon Nov 3 10:56:24 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 03 Nov 2014 17:56:24 +0100 Subject: [Nek5000-users] implement pressure and velocity BC Message-ID: Hi Neks, I'd like to use some specific function to define boundary conditions for pressure and velocity in my simulation. For example, I'd like to use for velocity Ux=Uy=0 dU/dz=a, and for pressure grad(p)=b. I've seen that in usercheck it is possible to write time dpendent functions to define the boundary conditions for velocity. But, is there any way to write Neumann boundary conditions for pressure or velocity? Or to write the function that I want? Thank you in advance. SL From nek5000-users at lists.mcs.anl.gov Mon Nov 3 14:13:11 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 3 Nov 2014 15:13:11 -0500 Subject: [Nek5000-users] grid-to-grid Message-ID: Hello all, I just wanted to use grid-to-grid interpolation tool as explained on the web to interpolate field data from a coarse resolution case to a high resolution mesh. But I have received the following runtime error, as shown in the following log file, when I run the case with the old geometry: . . . nsteps=0 -> skip time loop running solver in post processing mode call userchk grid-to-grid interpolation set initial conditions Checking restart options: old_fld0.f99999 nekuic (1) for ifld 1 nekuic (1) for ifld 2 Reading checkpoint data 0 OPEN: old_fld0.f99999 byte swap: F 6.543210030 -0.2931277218E+36 0 5.0000E+03 done :: Read checkpoint data avg data-throughput = 34.0MBps io-nodes = 1024 xyz min -3.2549 -14.105 -1.0850 uvwpt min -1.1888 -1.5496 -.73265 -.40517 .21138 PS min .00000E+00 .00000E+00 xyz max 3.2549 5.4249 1.0850 uvwpt max .79789 .84688 .61069 .94441 3.5997 PS max .00000E+00 .00000E+00 Restart: recompute geom. factors. regenerate geometry data 1 vol_t,vol_v: 274.173254616314068 274.173254616314068 done :: regenerate geometry data 1 done :: set initial conditions initializing intpts(), tol= 0.100000000000000003E-12 chunk 1 1 call intpts call findpts ERROR (proc 0000, ~/nek5_svn/trunk/nek/jl/findpts_imp.h:300): allocation of 3018752000 bytes failed I thought it might be due to the lpart parameter in the SIZE, but changing that did not solve the issue. The case was run on bgq machines with 1024 core with 1 GB/core. Here are the compilation info: ############################################################# # Compilation successful! # ############################################################# text data bss dec hex filename 13283121 1941456 37582536 52807113 325c5c9 nek5000 Note that there is no problem with normal running of both cases. I was wondering if anyone can help me on this issue. Thanks, Mohsen From nek5000-users at lists.mcs.anl.gov Wed Nov 5 08:33:18 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 05 Nov 2014 15:33:18 +0100 Subject: [Nek5000-users] implement pressure and velocity BC In-Reply-To: References: Message-ID: Hi Neks, I've seen that it is possible to impose BC to a scalar variable in the .rea file with ***** SCALAR 1 BOUNDARY CONDITIONS ***** I want to defie something like Ux=Uy=0, dUz/dx=b, grad(p)=a. If I impose v in ***** FLUID BOUNDARY CONDITIONS ***** I can then define Ux, Uy, Uz BC in the userbc subroutine. But, how can I define the BC in the pressure field? I've tried imposing outflow condition in ***** SCALAR 1 BOUNDARY CONDITIONS ***** Thus, I've copied all the BC defined in Fluid BC, and I've changed v by O. However, it doesn't work. What can I do? Is there any way to say Nek5000 that I'm going to define velocity and pressure BC in the userbc suroutine? Thank you. SL El 03-11-2014 17:56, nek5000-users at lists.mcs.anl.gov escribi?: > Hi Neks, > > I'd like to use some specific function to define boundary conditions > for pressure and velocity in my simulation. > > For example, I'd like to use for velocity Ux=Uy=0 dU/dz=a, and for > pressure grad(p)=b. I've seen that in usercheck it is possible to > write time dpendent functions to define the boundary conditions for > velocity. But, is there any way to write Neumann boundary conditions > for pressure or velocity? Or to write the function that I want? > > Thank you in advance. > SL > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Thu Nov 6 05:21:35 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 6 Nov 2014 12:21:35 +0100 Subject: [Nek5000-users] **ERROR**: Failed in HMHOLTZ Message-ID: Hi Neks when I setup a profile 2D flow, there are informations like this, but the simulation is not crash, Step 800, t= 8.0000000E-02, DT= 1.0000000E-04, C= 0.431 9.2444E+03 1.0117E+01 Solving for fluid F T T 800 100 **ERROR**: Failed in HMHOLTZ: VELX 3.7727E-16 5.3129E-02 1.0000E-16 800 100 **ERROR**: Failed in HMHOLTZ: VELY 2.4683E-16 3.5181E-02 1.0000E-16 800 U-PRES gmres: 23 5.6827E-15 1.0000E-14 9.6473E-07 9.5640E-01 2.4052E+00 800 8.0000E-02 1.0044E+01 Fluid done 800 8.00000000000E-02 6.70022578679E-01 1.99103625323E-01 4.70918953356E-01 1dragx 800 8.00000000000E-02 4.12612202513E-01 5.10664067531E-01 -9.80518650173E-02 1dragy dump history points done :: dump history points The CFL number is below than 0.5 already, the polynomial order is 7 in this case. when I choose a lower order p, this information will be disappeared after several iteration steps. I have checked the mesh looks fine, the parameters setting are copied from ?ext_cyl? with a smaller values of p21 to p25, I don?t know what is wrong, should I ignore ? thanks, Wei -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Thu Nov 6 05:28:23 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 6 Nov 2014 11:28:23 +0000 Subject: [Nek5000-users] **ERROR**: Failed in HMHOLTZ In-Reply-To: References: Message-ID: Hi Wei, Your Helmholtz tolerance of 1e-16 is too high (p22) -- velocity components start an interation from 5...4.e-2 residuals and converge to 4...2.e-16. Once you lower it, make sure the Divergence tolerance (p21) is still an order or 2 magnitude higher. Aleks ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Thursday, November 06, 2014 5:21 AM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] **ERROR**: Failed in HMHOLTZ Hi Neks when I setup a profile 2D flow, there are informations like this, but the simulation is not crash, Step 800, t= 8.0000000E-02, DT= 1.0000000E-04, C= 0.431 9.2444E+03 1.0117E+01 Solving for fluid F T T 800 100 **ERROR**: Failed in HMHOLTZ: VELX 3.7727E-16 5.3129E-02 1.0000E-16 800 100 **ERROR**: Failed in HMHOLTZ: VELY 2.4683E-16 3.5181E-02 1.0000E-16 800 U-PRES gmres: 23 5.6827E-15 1.0000E-14 9.6473E-07 9.5640E-01 2.4052E+00 800 8.0000E-02 1.0044E+01 Fluid done 800 8.00000000000E-02 6.70022578679E-01 1.99103625323E-01 4.70918953356E-01 1dragx 800 8.00000000000E-02 4.12612202513E-01 5.10664067531E-01 -9.80518650173E-02 1dragy dump history points done :: dump history points The CFL number is below than 0.5 already, the polynomial order is 7 in this case. when I choose a lower order p, this information will be disappeared after several iteration steps. I have checked the mesh looks fine, the parameters setting are copied from ?ext_cyl? with a smaller values of p21 to p25, I don?t know what is wrong, should I ignore ? thanks, Wei -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Thu Nov 6 05:31:30 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 6 Nov 2014 11:31:30 +0000 Subject: [Nek5000-users] **ERROR**: Failed in HMHOLTZ In-Reply-To: References: , Message-ID: P.S. Sorry, correction -- I meant you should increase your p22 from 1e-16 to. say 1.e-12 but you will also need to keep p21 at least 1 or 2 order of magnitude higher, say 1.e-10 Aleks ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Thursday, November 06, 2014 5:28 AM To: nek5000-users at lists.mcs.anl.gov Subject: Re: [Nek5000-users] **ERROR**: Failed in HMHOLTZ Hi Wei, Your Helmholtz tolerance of 1e-16 is too high (p22) -- velocity components start an interation from 5...4.e-2 residuals and converge to 4...2.e-16. Once you lower it, make sure the Divergence tolerance (p21) is still an order or 2 magnitude higher. Aleks ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Thursday, November 06, 2014 5:21 AM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] **ERROR**: Failed in HMHOLTZ Hi Neks when I setup a profile 2D flow, there are informations like this, but the simulation is not crash, Step 800, t= 8.0000000E-02, DT= 1.0000000E-04, C= 0.431 9.2444E+03 1.0117E+01 Solving for fluid F T T 800 100 **ERROR**: Failed in HMHOLTZ: VELX 3.7727E-16 5.3129E-02 1.0000E-16 800 100 **ERROR**: Failed in HMHOLTZ: VELY 2.4683E-16 3.5181E-02 1.0000E-16 800 U-PRES gmres: 23 5.6827E-15 1.0000E-14 9.6473E-07 9.5640E-01 2.4052E+00 800 8.0000E-02 1.0044E+01 Fluid done 800 8.00000000000E-02 6.70022578679E-01 1.99103625323E-01 4.70918953356E-01 1dragx 800 8.00000000000E-02 4.12612202513E-01 5.10664067531E-01 -9.80518650173E-02 1dragy dump history points done :: dump history points The CFL number is below than 0.5 already, the polynomial order is 7 in this case. when I choose a lower order p, this information will be disappeared after several iteration steps. I have checked the mesh looks fine, the parameters setting are copied from ?ext_cyl? with a smaller values of p21 to p25, I don?t know what is wrong, should I ignore ? thanks, Wei -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Thu Nov 6 05:48:04 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 6 Nov 2014 12:48:04 +0100 Subject: [Nek5000-users] **ERROR**: Failed in HMHOLTZ In-Reply-To: References: Message-ID: Hi Aleks, following your advice, it works. Thank you very much! Wei 2014-11-06 12:31 GMT+01:00 : > P.S. Sorry, correction -- I meant you should increase your p22 from > 1e-16 to. say 1.e-12 but you will also need to keep p21 at least 1 or 2 > order of magnitude higher, say 1.e-10 > > Aleks > ------------------------------ > *From:* nek5000-users-bounces at lists.mcs.anl.gov [ > nek5000-users-bounces at lists.mcs.anl.gov] on behalf of > nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > *Sent:* Thursday, November 06, 2014 5:28 AM > *To:* nek5000-users at lists.mcs.anl.gov > *Subject:* Re: [Nek5000-users] **ERROR**: Failed in HMHOLTZ > > Hi Wei, > > Your Helmholtz tolerance of 1e-16 is too high (p22) -- velocity components > start an interation from 5...4.e-2 residuals and converge to 4...2.e-16. > > Once you lower it, make sure the Divergence tolerance (p21) is still an > order or 2 magnitude higher. > > Aleks > ------------------------------ > *From:* nek5000-users-bounces at lists.mcs.anl.gov [ > nek5000-users-bounces at lists.mcs.anl.gov] on behalf of > nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > *Sent:* Thursday, November 06, 2014 5:21 AM > *To:* nek5000-users at lists.mcs.anl.gov > *Subject:* [Nek5000-users] **ERROR**: Failed in HMHOLTZ > > Hi Neks > > when I setup a profile 2D flow, there are informations like this, but > the simulation is not crash, > > Step 800, t= 8.0000000E-02, DT= 1.0000000E-04, C= 0.431 9.2444E+03 > 1.0117E+01 > Solving for fluid F T T > 800 100 **ERROR**: Failed in HMHOLTZ: VELX 3.7727E-16 5.3129E-02 > 1.0000E-16 > 800 100 **ERROR**: Failed in HMHOLTZ: VELY 2.4683E-16 3.5181E-02 > 1.0000E-16 > 800 U-PRES gmres: 23 5.6827E-15 1.0000E-14 9.6473E-07 > 9.5640E-01 2.4052E+00 > 800 8.0000E-02 1.0044E+01 Fluid done > 800 8.00000000000E-02 6.70022578679E-01 1.99103625323E-01 > 4.70918953356E-01 1dragx > 800 8.00000000000E-02 4.12612202513E-01 5.10664067531E-01 > -9.80518650173E-02 1dragy > dump history points > done :: dump history points > > > The CFL number is below than 0.5 already, the polynomial order is 7 in > this case. when I choose a lower order p, this information will be > disappeared after several iteration steps. I have checked the mesh looks > fine, the parameters setting are copied from ?ext_cyl? with a smaller > values of p21 to p25, I don?t know what is wrong, should I ignore ? > > thanks, > > Wei > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Thu Nov 6 13:52:01 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 6 Nov 2014 19:52:01 +0000 Subject: [Nek5000-users] Prex problem Message-ID: Hello, I have been struggling with this for a few days. I am just getting used to using nek5000. I am able to compile and run the examples on a multi-thread linux machine. However prex etc which includes a GUI opens up but I cannot see any text in the GUI window. They all appear as colored boxes. It actually took me a while to figure out what those colored boxes are. I have a feeling it's a compatibility issue. Any help would be appreciated. Thank you, -Eb -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Thu Nov 6 20:02:45 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Fri, 7 Nov 2014 02:02:45 +0000 Subject: [Nek5000-users] Prex problem In-Reply-To: References: Message-ID: Hi Eb, It's generally an X-windows compatibility issue... You might take a look at the makefile and see if there is an X11 reference that should be modified to match your system. Paul ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Thursday, November 06, 2014 1:52 PM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] Prex problem Hello, I have been struggling with this for a few days. I am just getting used to using nek5000. I am able to compile and run the examples on a multi-thread linux machine. However prex etc which includes a GUI opens up but I cannot see any text in the GUI window. They all appear as colored boxes. It actually took me a while to figure out what those colored boxes are. I have a feeling it?s a compatibility issue. Any help would be appreciated. Thank you, -Eb -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Fri Nov 7 09:13:10 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Fri, 7 Nov 2014 15:13:10 +0000 Subject: [Nek5000-users] Nekton (pre/post-processing) Message-ID: Hello, I am wondering if there's any manual for the nekton 2.6 out there. I have problem to work on with the pre-processing as menu feature provided by the existing manual is different with the one use in the version 2.6. Especially right at the beginning in the build menu (there's no ADD ELEMENT). Or perhaps is there any suggestion on other option of pre-processing. Thanks Best, Muhsin -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Fri Nov 7 09:42:58 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Fri, 7 Nov 2014 15:42:58 +0000 Subject: [Nek5000-users] Nekton (pre/post-processing) In-Reply-To: References: Message-ID: Dear Muhsin, The default is that it is always in "ADD ELEMENT" mode at the main build menu. So you simply click in 4 locations (preferably counterclockwise) and an element will build. Hour-glass or chevron configurations will not be generated so the newly entered 4 points will be ignored in those cases. Paul ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Friday, November 07, 2014 9:13 AM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] Nekton (pre/post-processing) Hello, I am wondering if there's any manual for the nekton 2.6 out there. I have problem to work on with the pre-processing as menu feature provided by the existing manual is different with the one use in the version 2.6. Especially right at the beginning in the build menu (there's no ADD ELEMENT). Or perhaps is there any suggestion on other option of pre-processing. Thanks Best, Muhsin -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Mon Nov 10 05:05:56 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 10 Nov 2014 06:05:56 -0500 Subject: [Nek5000-users] problem with running nek5000 on supercomputers Message-ID: Hi Neks, I am trying to run my simulation with nek5000 on supercomputers, the size of the problem is something like: text data bss dec hex filename 2233955 187412 2554984280 2557405647 986ee9cf nek5000 I don't have any problem when I use 32~64 CPUs to run parallel on supercomputers, but once I go for more than 64 CPUs, I always get "allocation failed" error. I have tried to run on different supercomputer like as Gordon SDSC, OSC, TACC, but I got the same type of error each time. I have checked the active memories, it seems for each of them memories have large unused portion. Does any one have any idea why when I increase the number of CPUs I get this allocation error?! Thank you, Ami -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Mon Nov 10 07:55:42 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 10 Nov 2014 13:55:42 +0000 Subject: [Nek5000-users] problem with running nek5000 on supercomputers In-Reply-To: References: Message-ID: Hi Ami, When you increase the processor count, you can recompile with a smaller lelt. Suppose you have lelt=1000 and you go from P=64 to P=128. Then set lelt=500 and recompile. Make certain lelg >= 500*P and lp >= 128. I would suggest lelg="big enough" (and an even number) and lp=32768. Paul ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Monday, November 10, 2014 5:05 AM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] problem with running nek5000 on supercomputers Hi Neks, I am trying to run my simulation with nek5000 on supercomputers, the size of the problem is something like: text data bss dec hex filename 2233955 187412 2554984280 2557405647 986ee9cf nek5000 I don't have any problem when I use 32~64 CPUs to run parallel on supercomputers, but once I go for more than 64 CPUs, I always get "allocation failed" error. I have tried to run on different supercomputer like as Gordon SDSC, OSC, TACC, but I got the same type of error each time. I have checked the active memories, it seems for each of them memories have large unused portion. Does any one have any idea why when I increase the number of CPUs I get this allocation error?! Thank you, Ami -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Mon Nov 10 15:29:19 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 10 Nov 2014 16:29:19 -0500 Subject: [Nek5000-users] problem with running nek5000 on supercomputers Message-ID: Dr. Fishcer, Thanks for your answer, for sure, I change the lelt and lelg and keep them as lowest even number that I can pick when I increase the number of CPUs. I picked also lp=32768, but I still get this "allocation failed" error. Is there any thing else that effect on the allocation which I have to change it?! Thanks, Ami On Mon, Nov 10, 2014 at 8:55 AM, wrote: > > Hi Ami, > > When you increase the processor count, you can recompile with a smaller > lelt. > > Suppose you have lelt=1000 and you go from P=64 to P=128. Then set > lelt=500 and > recompile. > > Make certain lelg >= 500*P and lp >= 128. > > I would suggest lelg="big enough" (and an even number) and lp=32768. > > Paul > > ------------------------------ > *From:* nek5000-users-bounces at lists.mcs.anl.gov [ > nek5000-users-bounces at lists.mcs.anl.gov] on behalf of > nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > *Sent:* Monday, November 10, 2014 5:05 AM > *To:* nek5000-users at lists.mcs.anl.gov > *Subject:* [Nek5000-users] problem with running nek5000 on supercomputers > > Hi Neks, > > I am trying to run my simulation with nek5000 on supercomputers, the size > of the problem is something like: > > text data bss dec hex filename > 2233955 187412 2554984280 2557405647 986ee9cf nek5000 > > I don't have any problem when I use 32~64 CPUs to run parallel on > supercomputers, but once I go for more than 64 CPUs, I always get > "allocation failed" error. I have tried to run on different supercomputer > like as Gordon SDSC, OSC, TACC, but I got the same type of error each time. > I have checked the active memories, it seems for each of them memories have > large unused portion. Does any one have any idea why when I increase the > number of CPUs I get this allocation error?! > > Thank you, > Ami > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Tue Nov 11 03:23:54 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 11 Nov 2014 04:23:54 -0500 Subject: [Nek5000-users] problem with running nek5000 on supercomputers Message-ID: It seems that for each time I had to clean compiled files before changing the processor count and run it again! It works now! Thanks, Ami On Mon, Nov 10, 2014 at 4:29 PM, Amirreza Hashemi wrote: > Dr. Fishcer, > > Thanks for your answer, for sure, I change the lelt and lelg and keep them > as lowest even number that I can pick when I increase the number of CPUs. I > picked also lp=32768, but I still get this "allocation failed" error. Is > there any thing else that effect on the allocation which I have to change > it?! > > Thanks, > Ami > > On Mon, Nov 10, 2014 at 8:55 AM, wrote: > >> >> Hi Ami, >> >> When you increase the processor count, you can recompile with a smaller >> lelt. >> >> Suppose you have lelt=1000 and you go from P=64 to P=128. Then set >> lelt=500 and >> recompile. >> >> Make certain lelg >= 500*P and lp >= 128. >> >> I would suggest lelg="big enough" (and an even number) and lp=32768. >> >> Paul >> >> ------------------------------ >> *From:* nek5000-users-bounces at lists.mcs.anl.gov [ >> nek5000-users-bounces at lists.mcs.anl.gov] on behalf of >> nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >> *Sent:* Monday, November 10, 2014 5:05 AM >> *To:* nek5000-users at lists.mcs.anl.gov >> *Subject:* [Nek5000-users] problem with running nek5000 on supercomputers >> >> Hi Neks, >> >> I am trying to run my simulation with nek5000 on supercomputers, the >> size of the problem is something like: >> >> text data bss dec hex filename >> 2233955 187412 2554984280 2557405647 986ee9cf nek5000 >> >> I don't have any problem when I use 32~64 CPUs to run parallel on >> supercomputers, but once I go for more than 64 CPUs, I always get >> "allocation failed" error. I have tried to run on different supercomputer >> like as Gordon SDSC, OSC, TACC, but I got the same type of error each time. >> I have checked the active memories, it seems for each of them memories have >> large unused portion. Does any one have any idea why when I increase the >> number of CPUs I get this allocation error?! >> >> Thank you, >> Ami >> >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Tue Nov 11 07:25:01 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 11 Nov 2014 13:25:01 +0000 Subject: [Nek5000-users] problem with running nek5000 on supercomputers In-Reply-To: References: Message-ID: Hi Ami, Great to hear the problem is fixed! In principle, it should suffice to simply makenek blah after editing the SIZE file, since make is smart enough to know that you've touched SIZE. Normally I don't recompile when changing processor count (unless it's by a huge factor). So, suppose my job has 4000 elements and I'm running on anywhere from 16 to 2048 processors. It's sufficient to set lelt=(4000/16) and lp=2048. Then you can run on that range P=16 to 2048 without recompile. Paul ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Tuesday, November 11, 2014 3:23 AM To: nek5000-users at lists.mcs.anl.gov Subject: Re: [Nek5000-users] problem with running nek5000 on supercomputers It seems that for each time I had to clean compiled files before changing the processor count and run it again! It works now! Thanks, Ami On Mon, Nov 10, 2014 at 4:29 PM, Amirreza Hashemi > wrote: Dr. Fishcer, Thanks for your answer, for sure, I change the lelt and lelg and keep them as lowest even number that I can pick when I increase the number of CPUs. I picked also lp=32768, but I still get this "allocation failed" error. Is there any thing else that effect on the allocation which I have to change it?! Thanks, Ami On Mon, Nov 10, 2014 at 8:55 AM, > wrote: Hi Ami, When you increase the processor count, you can recompile with a smaller lelt. Suppose you have lelt=1000 and you go from P=64 to P=128. Then set lelt=500 and recompile. Make certain lelg >= 500*P and lp >= 128. I would suggest lelg="big enough" (and an even number) and lp=32768. Paul ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Monday, November 10, 2014 5:05 AM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] problem with running nek5000 on supercomputers Hi Neks, I am trying to run my simulation with nek5000 on supercomputers, the size of the problem is something like: text data bss dec hex filename 2233955 187412 2554984280 2557405647 986ee9cf nek5000 I don't have any problem when I use 32~64 CPUs to run parallel on supercomputers, but once I go for more than 64 CPUs, I always get "allocation failed" error. I have tried to run on different supercomputer like as Gordon SDSC, OSC, TACC, but I got the same type of error each time. I have checked the active memories, it seems for each of them memories have large unused portion. Does any one have any idea why when I increase the number of CPUs I get this allocation error?! Thank you, Ami _______________________________________________ Nek5000-users mailing list Nek5000-users at lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Tue Nov 11 07:33:22 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 11 Nov 2014 08:33:22 -0500 Subject: [Nek5000-users] problem with running nek5000 on supercomputers In-Reply-To: References: Message-ID: Thank you Dr. Fischer, I think it became so clear, maybe the problem for me happened since I first ran nek with 32 processors and then I went to run it on 1024! Which I found it little bit problematic on couple of cases that I have tested! Thank you, Ami On Tue, Nov 11, 2014 at 8:25 AM, wrote: > > Hi Ami, > > Great to hear the problem is fixed! > > In principle, it should suffice to simply > > makenek blah > > after editing the SIZE file, since make is smart enough to know that > you've touched SIZE. > > Normally I don't recompile when changing processor count (unless it's by > a huge factor). > > So, suppose my job has 4000 elements and I'm running on anywhere from 16 > to 2048 > processors. It's sufficient to set lelt=(4000/16) and lp=2048. Then you > can run on that > range P=16 to 2048 without recompile. > > Paul > > > ------------------------------ > *From:* nek5000-users-bounces at lists.mcs.anl.gov [ > nek5000-users-bounces at lists.mcs.anl.gov] on behalf of > nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > *Sent:* Tuesday, November 11, 2014 3:23 AM > *To:* nek5000-users at lists.mcs.anl.gov > *Subject:* Re: [Nek5000-users] problem with running nek5000 on > supercomputers > > It seems that for each time I had to clean compiled files before > changing the processor count and run it again! It works now! > > Thanks, > Ami > > On Mon, Nov 10, 2014 at 4:29 PM, Amirreza Hashemi < > amirezahashemi at gmail.com> wrote: > >> Dr. Fishcer, >> >> Thanks for your answer, for sure, I change the lelt and lelg and keep >> them as lowest even number that I can pick when I increase the number of >> CPUs. I picked also lp=32768, but I still get this "allocation failed" >> error. Is there any thing else that effect on the allocation which I have >> to change it?! >> >> Thanks, >> Ami >> >> On Mon, Nov 10, 2014 at 8:55 AM, >> wrote: >> >>> >>> Hi Ami, >>> >>> When you increase the processor count, you can recompile with a >>> smaller lelt. >>> >>> Suppose you have lelt=1000 and you go from P=64 to P=128. Then set >>> lelt=500 and >>> recompile. >>> >>> Make certain lelg >= 500*P and lp >= 128. >>> >>> I would suggest lelg="big enough" (and an even number) and lp=32768. >>> >>> Paul >>> >>> ------------------------------ >>> *From:* nek5000-users-bounces at lists.mcs.anl.gov [ >>> nek5000-users-bounces at lists.mcs.anl.gov] on behalf of >>> nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >>> *Sent:* Monday, November 10, 2014 5:05 AM >>> *To:* nek5000-users at lists.mcs.anl.gov >>> *Subject:* [Nek5000-users] problem with running nek5000 on >>> supercomputers >>> >>> Hi Neks, >>> >>> I am trying to run my simulation with nek5000 on supercomputers, the >>> size of the problem is something like: >>> >>> text data bss dec hex filename >>> 2233955 187412 2554984280 2557405647 986ee9cf nek5000 >>> >>> I don't have any problem when I use 32~64 CPUs to run parallel on >>> supercomputers, but once I go for more than 64 CPUs, I always get >>> "allocation failed" error. I have tried to run on different supercomputer >>> like as Gordon SDSC, OSC, TACC, but I got the same type of error each time. >>> I have checked the active memories, it seems for each of them memories have >>> large unused portion. Does any one have any idea why when I increase the >>> number of CPUs I get this allocation error?! >>> >>> Thank you, >>> Ami >>> >>> _______________________________________________ >>> Nek5000-users mailing list >>> Nek5000-users at lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>> >>> >> > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Tue Nov 11 13:03:19 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 11 Nov 2014 19:03:19 +0000 Subject: [Nek5000-users] Nekton (pre/post-processing) In-Reply-To: References: Message-ID: Hi Dr Fischer, Thank you for your reply. I would like to add some question, I wonder if there's any command to abort a command such when you've mistaken clicked the [modify element] command or perhaps to undo it? This is because I usually delete an element or restart the prenek whenever I've mistaken created/modify an element as I not . Plus, I wonder of how can I obtain nekton 2.8, since at the moment I'm using nekton 2.6 and the feature for its menu are pretty different from the manual available online. Thank you. Best regards, Muhsin On 7 November 2014 15:42, wrote: > > Dear Muhsin, > > The default is that it is always in "ADD ELEMENT" mode at the main build > menu. > > So you simply click in 4 locations (preferably counterclockwise) and an > element will build. > > Hour-glass or chevron configurations will not be generated so the newly > entered 4 points > will be ignored in those cases. > > Paul > > ------------------------------ > *From:* nek5000-users-bounces at lists.mcs.anl.gov [ > nek5000-users-bounces at lists.mcs.anl.gov] on behalf of > nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > *Sent:* Friday, November 07, 2014 9:13 AM > *To:* nek5000-users at lists.mcs.anl.gov > *Subject:* [Nek5000-users] Nekton (pre/post-processing) > > Hello, > > I am wondering if there's any manual for the nekton 2.6 out there. I have > problem to work on with the pre-processing as menu feature provided by the > existing manual is different with the one use in the version 2.6. > Especially right at the beginning in the build menu (there's no ADD > ELEMENT). Or perhaps is there any suggestion on other option of > pre-processing. Thanks > > Best, > Muhsin > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Thu Nov 13 08:52:23 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 13 Nov 2014 14:52:23 +0000 Subject: [Nek5000-users] Recycling BC Message-ID: Dear Neks, I am using the turbJet example to set up my own recycling boundary case. As far as I understand the velocity that is passing through stack 8 (element 8) in the turbJet example should then be recycled back to the inlet. However when I plot the instantaneous velocity profiles for these two points they are non-identical. Is this deliberately so, I presume it should behave like a periodic boundary condition, or have I missed the point? Thank you for your time, Friedrich. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Mon Nov 24 21:44:13 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 24 Nov 2014 22:44:13 -0500 Subject: [Nek5000-users] IO problem Message-ID: Hello all, I have sometimes experienced an IO issue, when running on a relatively large number of machines on our bgq cluster with MPIIO enabled and p65=0. The issue is that while the status of the job shows normal running, no output files and even .log file are written on the file system. What is strange is that the issue happens occasionally and for jobs with more than 512 nodes, as I haven't seen this for smaller scale problems and resources. Usually, cancelling and resubmitting the job will solve the issue. I contacted the technical staff of our cluster and they performed several tests but they have not yet been able to reproduce the problem on small scales. It is more likely that the this problem is caused by a hardware issue rather than a code issue since it is not happened all the time. However, I wanted to contact you and wondered if anyone has any comments or similar experience. Thanks Mohsen