From nek5000-users at lists.mcs.anl.gov Fri Oct 3 05:51:43 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Fri, 3 Oct 2014 10:51:43 +0000 Subject: [Nek5000-users] Derivatives problem Message-ID: Dear Paul, Sorry for the very late reply but I was carrying out various checks to see where the problem was coming from. It seems that there are some strange values at the boundaries of the spectral elements in the instantaneous fields (i.e. in the original nek dumps prior to any post-processing). We are using Visit 2.7.0. The problem is evident when the data limits are changed. Some examples: x-velocity, zoomed in, default data limits: https://www.dropbox.com/s/ewrlwyjqmg8kfx4/L1_xvel_slice_zoom_default_range.png?dl=0 x-velocity, zoomed in, reduced data limits: https://www.dropbox.com/s/1wy8j6mu4z20lyf/L2_xvel_slice_zoom_reduced_range.png?dl=0 same as above with superimposed mesh: https://www.dropbox.com/s/mcungp7vr2un3da/L3_xvel_slice_zoom_reduced_range_w_mesh.png?dl=0 Similar results are obtained with the y-vel, z-vel and pressure. We have also carried out statistics on the above (this is where we started noticing about the problem at the spectral element boundaries). We imagine the problems we see are directly linked to the above. Some results, when averaged, seem to make the problem disappear whilst others still evidently have the problem: (note that the following are Paraview visualizations) averaged x-velocity (over time and in the z/spanwise/homogeneous direction) , zoomed in, default data limits: https://www.dropbox.com/s/rbvhpfcth4hst96/L4_STAT_avg_x_zoom_default_range.png?dl=0 (even when changing the data limits we could not spot any strange values at the boundaries) Turbulent kinetic energy - default data limits: https://www.dropbox.com/s/529cdukvl8t86va/L5_TKE.png?dl=0 (even when changing the data limits we could not spot any strange values at the boundaries) Problem - Production - default data limits: https://www.dropbox.com/s/qyygzzmg3r3fbve/L6_PROD.png?dl=0 Problem - Dissipation - default data limits: https://www.dropbox.com/s/q27097z5wsi57a5/L7_DISSIPATION.png?dl=0 For a less resolved case, the problem also becomes evident in, for example, the averaged velocities and averaged squared fluctuations. The problem is not seen for these statistics in the well resolved case BUT is still seen (as shown above) for the Production case... At least we imagine that the problem is not due to mesh resolution... We thought it was a problem in our post-processing script but since we see this problem in the instantaneous dumps too we are a bit confused! Some details about the simulation: Channel flow with a lower curved wall, periodic boundary conditions in the x-direction (streamwise), periodic boundary conditions in the z-direction (spanwise) and top and bottom walls. Initial condition is zero and a forcing is imposed in the x-direction to sustain flow. Re=10000 in .rea. Find .box and SIZE files in the link below. We used a routine in usrdat2() to modify the geometry and create a bump. You can find the .box and SIZE files attached here: http://we.tl/slw0QxA6wr Let me know if you need any more information.. Thank you very much for your help and time, JP -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Fri Oct 3 14:36:10 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Fri, 3 Oct 2014 13:36:10 -0600 Subject: [Nek5000-users] Moving Wall Advice Message-ID: Hello Neks, I'm trying to simulate a flow where one of the boundaries is moving in a manner that requires the mesh to deform as a function of time. In short, the conditions of the flow will determine how the boundary moves (it's a fluid-structure interaction problem, but the structure is a rigid body suspended in the fluid). Are there any available examples that would provide guidance? If not, does anyone out there want to share some wisdom? For example, I understand that I will need IFMVBD = True, and I will need "mv" boundary conditions specified in the .rea file, but in which routine should I modify the mesh node locations and velocities? I found a similar question in the archive under the subject "what parameters for a moving wall induced velocity problem?" but there didn't seem to be any follow-up responses. Thanks in advance. Cheers, Mike From nek5000-users at lists.mcs.anl.gov Tue Oct 7 08:34:28 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 7 Oct 2014 07:34:28 -0600 Subject: [Nek5000-users] Moving Wall Advice In-Reply-To: References: Message-ID: Hello all, I have not seen any response on this -- anyone have some thoughts they'd like to share? Cheers, Mike On Oct 3, 2014, at 1:36 PM, wrote: > Hello Neks, > > I'm trying to simulate a flow where one of the boundaries is moving in a manner that requires the mesh to deform as a function of time. In short, the conditions of the flow will determine how the boundary moves (it's a fluid-structure interaction problem, but the structure is a rigid body suspended in the fluid). > > Are there any available examples that would provide guidance? > > If not, does anyone out there want to share some wisdom? For example, I understand that I will need IFMVBD = True, and I will need "mv" boundary conditions specified in the .rea file, but in which routine should I modify the mesh node locations and velocities? > > I found a similar question in the archive under the subject "what parameters for a moving wall induced velocity problem?" but there didn't seem to be any follow-up responses. > > Thanks in advance. > > Cheers, > Mike > > > > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Tue Oct 7 09:00:06 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 7 Oct 2014 14:00:06 +0000 Subject: [Nek5000-users] Moving Wall Advice In-Reply-To: References: , Message-ID: Hi Mike, A grep in the examples folder under the Nek repository gave me the example 'fs_hydro' as a comparable case. I'd also take a look at the primer http://www.mcs.anl.gov/~fischer/primer.pdf in particular page 16. You would probably need different boundary conditions since these cases are free surface cases but maybe you can tweak something to get started. I am not sure if time dependency for the boundary is covered in any of these examples but you can implement the time advancement in the user file using the same time step as the solver. Good luck and let us know how it goes, Oana ________________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Tuesday, October 07, 2014 8:34 AM To: nek5000-users at lists.mcs.anl.gov Subject: Re: [Nek5000-users] Moving Wall Advice Hello all, I have not seen any response on this -- anyone have some thoughts they'd like to share? Cheers, Mike On Oct 3, 2014, at 1:36 PM, wrote: > Hello Neks, > > I'm trying to simulate a flow where one of the boundaries is moving in a manner that requires the mesh to deform as a function of time. In short, the conditions of the flow will determine how the boundary moves (it's a fluid-structure interaction problem, but the structure is a rigid body suspended in the fluid). > > Are there any available examples that would provide guidance? > > If not, does anyone out there want to share some wisdom? For example, I understand that I will need IFMVBD = True, and I will need "mv" boundary conditions specified in the .rea file, but in which routine should I modify the mesh node locations and velocities? > > I found a similar question in the archive under the subject "what parameters for a moving wall induced velocity problem?" but there didn't seem to be any follow-up responses. > > Thanks in advance. > > Cheers, > Mike > > > > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users _______________________________________________ Nek5000-users mailing list Nek5000-users at lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Tue Oct 7 09:08:24 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 7 Oct 2014 14:08:24 +0000 Subject: [Nek5000-users] Moving Wall Advice In-Reply-To: References: Message-ID: Hi Mike, There is also the ocyl example, which has an oscillating cylinder. Paul ________________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Friday, October 03, 2014 2:36 PM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] Moving Wall Advice Hello Neks, I'm trying to simulate a flow where one of the boundaries is moving in a manner that requires the mesh to deform as a function of time. In short, the conditions of the flow will determine how the boundary moves (it's a fluid-structure interaction problem, but the structure is a rigid body suspended in the fluid). Are there any available examples that would provide guidance? If not, does anyone out there want to share some wisdom? For example, I understand that I will need IFMVBD = True, and I will need "mv" boundary conditions specified in the .rea file, but in which routine should I modify the mesh node locations and velocities? I found a similar question in the archive under the subject "what parameters for a moving wall induced velocity problem?" but there didn't seem to be any follow-up responses. Thanks in advance. Cheers, Mike _______________________________________________ Nek5000-users mailing list Nek5000-users at lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Tue Oct 7 09:31:16 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 7 Oct 2014 08:31:16 -0600 Subject: [Nek5000-users] Moving Wall Advice In-Reply-To: References: Message-ID: Paul, Oana -- thanks for the tips. The ocyl example seems very close to what I need. One question: Will the ALE formulation as currently implemented work if I'm solving a 2D axisymmetric-flow problem (where the temperature is used to represent the azimuthal flow component)? Thanks again, Mike On Oct 7, 2014, at 8:08 AM, wrote: > > Hi Mike, > > There is also the ocyl example, which has an oscillating cylinder. > > Paul > > ________________________________________ > From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > Sent: Friday, October 03, 2014 2:36 PM > To: nek5000-users at lists.mcs.anl.gov > Subject: [Nek5000-users] Moving Wall Advice > > Hello Neks, > > I'm trying to simulate a flow where one of the boundaries is moving in a manner that requires the mesh to deform as a function of time. In short, the conditions of the flow will determine how the boundary moves (it's a fluid-structure interaction problem, but the structure is a rigid body suspended in the fluid). > > Are there any available examples that would provide guidance? > > If not, does anyone out there want to share some wisdom? For example, I understand that I will need IFMVBD = True, and I will need "mv" boundary conditions specified in the .rea file, but in which routine should I modify the mesh node locations and velocities? > > I found a similar question in the archive under the subject "what parameters for a moving wall induced velocity problem?" but there didn't seem to be any follow-up responses. > > Thanks in advance. > > Cheers, > Mike > > > > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Tue Oct 7 09:53:48 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 7 Oct 2014 14:53:48 +0000 Subject: [Nek5000-users] Moving Wall Advice In-Reply-To: References: , Message-ID: Hi Mike, No idea of whether it will work or not...I would suggest to try it and verify (which is what I would do in that case). In principle, it should work ok, but I don't know that anyone has tried it for a very long time. Note that the SIZE file for moving mesh is different from the usual one. Paul ________________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Tuesday, October 07, 2014 9:31 AM To: nek5000-users at lists.mcs.anl.gov Subject: Re: [Nek5000-users] Moving Wall Advice Paul, Oana -- thanks for the tips. The ocyl example seems very close to what I need. One question: Will the ALE formulation as currently implemented work if I'm solving a 2D axisymmetric-flow problem (where the temperature is used to represent the azimuthal flow component)? Thanks again, Mike On Oct 7, 2014, at 8:08 AM, wrote: > > Hi Mike, > > There is also the ocyl example, which has an oscillating cylinder. > > Paul > > ________________________________________ > From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > Sent: Friday, October 03, 2014 2:36 PM > To: nek5000-users at lists.mcs.anl.gov > Subject: [Nek5000-users] Moving Wall Advice > > Hello Neks, > > I'm trying to simulate a flow where one of the boundaries is moving in a manner that requires the mesh to deform as a function of time. In short, the conditions of the flow will determine how the boundary moves (it's a fluid-structure interaction problem, but the structure is a rigid body suspended in the fluid). > > Are there any available examples that would provide guidance? > > If not, does anyone out there want to share some wisdom? For example, I understand that I will need IFMVBD = True, and I will need "mv" boundary conditions specified in the .rea file, but in which routine should I modify the mesh node locations and velocities? > > I found a similar question in the archive under the subject "what parameters for a moving wall induced velocity problem?" but there didn't seem to be any follow-up responses. > > Thanks in advance. > > Cheers, > Mike > > > > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users _______________________________________________ Nek5000-users mailing list Nek5000-users at lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Wed Oct 8 12:17:48 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 8 Oct 2014 10:17:48 -0700 Subject: [Nek5000-users] Upper Memory Bounds per Core Message-ID: Neks, I was attempting to run a scaled down problem on my workstation to prepare for submission on a cluster when I noticed some strange behavior. The scaled down version of the problem has 1716 elements and I am running it with lx1=8. According to Dr. Fischer's estimate in the primer this should be around 4 GB of memory, and my workstation has 32 GB of memory so it shouldn't be a problem. However, when I attempt to compile in serial (lp=1, lelt=1716) makenek aborts telling me there is not enough memory available. When I compile with lp>1 and lelt<1100 it will compile and run without a problem. I found that the real bottle neck is that lelt has to be below 1100 for this particular problem. This leads to my question: how does Nek determine if there is enough memory during compilation and can it be modified? Is there a hardcoded value somewhere in the source code? I scanned through the SIZE file and makenek script, but nothing popped out at me. Thanks, -- Phil Sakievich PhD Student - Mechanical Engineering Arizona State University - Ira A. Fulton School for Engineering of Matter Transport and Energy Tempe, Arizona -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Wed Oct 8 12:21:28 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 8 Oct 2014 19:21:28 +0200 Subject: [Nek5000-users] Upper Memory Bounds per Core In-Reply-To: References: Message-ID: Hi, Any chance you would be compiling using gfortran? I had a similar problem back then and could solve it simply by adding -mcmodel=medium to the compilation options. ++ JC -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Thu Oct 9 10:43:51 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 9 Oct 2014 08:43:51 -0700 Subject: [Nek5000-users] Upper Memory Bounds per Core Message-ID: JC, I am using gfortran, and yes I believe that fixed it. Thanks! -- Phil Sakievich PhD Student - Mechanical Engineering Arizona State University - Ira A. Fulton School for Engineering of Matter Transport and Energy Tempe, Arizona -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Thu Oct 9 11:27:46 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 9 Oct 2014 16:27:46 +0000 Subject: [Nek5000-users] Moab Issues Message-ID: Hi All, I am trying to use moab enabled Nek5000 and have followed all steps installing and linking moab, HDF5 and Zoltan together successfully. However when I try to run the moab example, mpif77 -c -O2 -r8 -fpconstant -fpp -DMOAB -DPTRSIZE8 -DMPI -DLONGINT8 -DUNDERSCORE -DGLOBAL_LONG_LONG -I/home/eng/esumgy/nek5_svn/examples/moab -I/home/eng/esumgy/nek5_svn/trunk/nek -I./ -DUNORDERED_MAP_NS=std::tr1 -DHAVE_UNORDERED_MAP=tr1/unordered_map -DHAVE_UNORDERED_SET=tr1/unordered_set -I/home/eng/esumgy/hdf5/hdf5-1.8/hdf5/include -isystem /home/eng/esumgy/hdf5/hdf5-1.8/hdf5/include -DTEMPLATE_SPECIALIZATION -DTEMPLATE_FUNC_SPECIALIZATION -DHAVE_VSNPRINTF -D_FILE_OFFSET_BITS=64 -DHDF5_FILE -I/home/eng/esumgy/moab-4.6.3/include /home/eng/esumgy/nek5_svn/trunk/nek/3rd_party/moab.f -o obj/moab.o MOABCORE(16): #error: can't find include file: iMeshP_f.h MOABCORE(16): #error: can't find include file: iMeshP_f.h MOABCORE(16): #error: can't find include file: iMeshP_f.h MOABCORE(16): #error: can't find include file: iMeshP_f.h MOABCORE(16): #error: can't find include file: iMeshP_f.h MOABCORE(16): #error: can't find include file: iMeshP_f.h MOABCORE(16): #error: can't find include file: iMeshP_f.h MOABCORE(16): #error: can't find include file: iMeshP_f.h moab.f(545): #error: can't find include file: iMeshP_f.h moab.f(720): #error: can't find include file: iMeshP_f.h MOABCORE(16): #error: can't find include file: iMeshP_f.h And so on... The iMesh part I believe is linked to the definition of the Moab directory in the makenek file, which I have set up accordingly so I am at a loss why it is finding the file. I thought it may be a problem with the hdf5 installation which is currently just set up in series not parallel so I have tried to play with the installation options here, but this does not effect the outcome. Thanks, Friedrich. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Thu Oct 9 11:31:02 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 9 Oct 2014 11:31:02 -0500 Subject: [Nek5000-users] Moab Issues In-Reply-To: References: Message-ID: Friedrich, Clearly its not finding the iMeshP_f.h includes. Can you verify whether the file exists in your installation at /home/eng/esumgy/moab-4.6.3/include ? If not, you will have to re-configure MOAB with --enable-imesh and install it again. If possible, please also upgrade to 4.7 since we had quite a few fixes and enhancements in the newer versions. Vijay On Thu, Oct 9, 2014 at 11:27 AM, wrote: > Hi All, > > > I am trying to use moab enabled Nek5000 and have followed all steps > installing and linking moab, HDF5 and Zoltan together successfully. > > > However when I try to run the moab example, > > > mpif77 -c -O2 -r8 -fpconstant -fpp -DMOAB -DPTRSIZE8 -DMPI -DLONGINT8 > -DUNDERSCORE -DGLOBAL_LONG_LONG -I/home/eng/esumgy/nek5_svn/examples/moab > -I/home/eng/esumgy/nek5_svn/trunk/nek -I./ -DUNORDERED_MAP_NS=std::tr1 > -DHAVE_UNORDERED_MAP=tr1/unordered_map > -DHAVE_UNORDERED_SET=tr1/unordered_set > -I/home/eng/esumgy/hdf5/hdf5-1.8/hdf5/include -isystem > /home/eng/esumgy/hdf5/hdf5-1.8/hdf5/include -DTEMPLATE_SPECIALIZATION > -DTEMPLATE_FUNC_SPECIALIZATION -DHAVE_VSNPRINTF -D_FILE_OFFSET_BITS=64 > -DHDF5_FILE -I/home/eng/esumgy/moab-4.6.3/include > /home/eng/esumgy/nek5_svn/trunk/nek/3rd_party/moab.f -o obj/moab.o > MOABCORE(16): #error: can't find include file: iMeshP_f.h > MOABCORE(16): #error: can't find include file: iMeshP_f.h > MOABCORE(16): #error: can't find include file: iMeshP_f.h > MOABCORE(16): #error: can't find include file: iMeshP_f.h > MOABCORE(16): #error: can't find include file: iMeshP_f.h > MOABCORE(16): #error: can't find include file: iMeshP_f.h > MOABCORE(16): #error: can't find include file: iMeshP_f.h > MOABCORE(16): #error: can't find include file: iMeshP_f.h > moab.f(545): #error: can't find include file: iMeshP_f.h > moab.f(720): #error: can't find include file: iMeshP_f.h > MOABCORE(16): #error: can't find include file: iMeshP_f.h > > > And so on... > > > The iMesh part I believe is linked to the definition of the Moab directory > in the makenek file, which I have set up accordingly so I am at a loss why > it is finding the file. > > > I thought it may be a problem with the hdf5 installation which is currently > just set up in series not parallel so I have tried to play with the > installation options here, but this does not effect the outcome. > > > Thanks, > > > Friedrich. > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > From nek5000-users at lists.mcs.anl.gov Sun Oct 12 06:12:29 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sun, 12 Oct 2014 11:12:29 +0000 Subject: [Nek5000-users] Moab Issues Message-ID: Hi All, Thanks for your response Vijay, I managed to get both moab examples to work successfully! (As an aside I found my simulation works the same with or without Zoltan thus is there a point in having it installed?) However my main issue is now that when I try to use my own .h5m mesh my simulation crashes citing following error: read .rea file ASSERT ERROR: 14 in moab.f line 477 This error has been previously flagged and the solution was given that I must first partition my mesh into parallel sections using: mpiexec -np 4 mbconvert -O PARALLEL=READ_PART -O PARTITION=PARALLEL_PARTITION -O PARALLEL_RESOLVE_SHARED_ENTS -O PARALLEL_GHOSTS=3.0.1 -o PARALLEL=WRITE_PART dummy.h5m In true fashion I encounter further errors! When running this function using my mesh created using Cubit (Trelis) I receive the error: Failed to load "channel.h5m". Error code: MB_NOT_IMPLEMENTED (9) Error message: Failed in step PARALLEL READ PART NULL file handle. This error is also displayed when I attempt to run the test using the example listed on the Moab webpage: http://trac.mcs.anl.gov/projects/ITAPS/wiki/ParallelRead I understand this may not be your area of expertise but any help is appreciated! Maybe my approach to the creation of new meshes is incorrect. I am using Cubit(Trelis) to create my mesh and set my boundary conditions then converting from .cub to .h5m using mbconvert. Then reading using Moab enable Nek5000. I am intending to eventually run jet flow simulations and feel this method is easier to create the geometry than using prex or any of the other tools especially when using a fairly complex and changing mesh geometry? Thank you for your time, Friedrich Grabner. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Mon Oct 13 11:55:58 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 13 Oct 2014 16:55:58 +0000 Subject: [Nek5000-users] Moab Issues Message-ID: Hi All, Sorry to post so many questions but I am tearing my hair out! I solved my previous issue just a matter of installing the Zoltan tools and using them to partition my mesh instead. However I have now another obstacle to overcome. I keep getting the error message: Parallel Read times: 0.0771379 PARALLEL READ PART 0.00908303 PARALLEL RESOLVE_SHARED_ENTS 0.000237942 PARALLEL RESOLVE_SHARED_SETS 0.0863409 PARALLEL TOTAL ABORT: nelv is invalid in nekmoab_proc_map ABORT: nelv is invalid in nekmoab_proc_map nelv, lelv = 0 64 ABORT: nelv is invalid in nekmoab_proc_map nelv, lelv = 0 64 nelv, lelv = 0 64 ABORT: nelv is invalid in nekmoab_proc_map nelv, lelv = 0 64 call exitt: dying ... This has been asked before and the problem was identified that moab enabled Nek5000 is not compatible with 2D quads. However I have a 3D mesh, proven by: type count total ------------------------------ Edge 282 3.7e+02 Quad 152 2.6e+02 Hex 64 1.2e+02 1D Side 1376 1.8e+03 Vertex 765 A very small mesh I know. So the issue seems to be that nelv is less than 64 and in the moab.f routine this brings the above error (I've tried simply commenting this out unsuccessfully). Thus it would seem logical that one defines nelv to be the correct value. I have tried this by defining it in the geometry section of the .rea file and this brings even more errors. To my knowledge as my mesh is 3D this subroutine shouldn't even be called. Any help is appreciated! Thanks for your time, Friedrich. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Tue Oct 14 06:55:15 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 14 Oct 2014 11:55:15 +0000 Subject: [Nek5000-users] Moab Issues In-Reply-To: References: Message-ID: Hi Friedriech, It appears that Nek-MOAB does not pickup your elements since nelv=0. One thing to check is whether your block-ids/material sets are specified correctly in the mesh files .cub & .h5m and in the .rea file -- e.g. the following lines in nek5_svn/examples/moab_conjht/moab_conjht.rea: 1 2 1 fluid set, 2 other/solid sets 20 10 30 fluid set #20, solid sets #10, 30 correspond to a Cubit/MOAB mesh with 3 block-ids/material sets: 10,20, and 30 which matches the check with 'mbsize -m' below. If the problem still persists, please contact me offline at obabko at mcs.nal.gov Aleks mbsize -m moab_conjht.h5m ... Material Set 10: type count total minimum average rms maximum std.dev. ------- ----- ------- ------------------------- ------------------------- ------------------------- ------------------------- ------------------------- Hex 1 1 1 1 1 1 0 1D Side 12 12 1 1 1 1 0 Vertex 0 Material Set 20: .... Material Set 30: ... ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Monday, October 13, 2014 11:55 AM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] Moab Issues Hi All, Sorry to post so many questions but I am tearing my hair out! I solved my previous issue just a matter of installing the Zoltan tools and using them to partition my mesh instead. However I have now another obstacle to overcome. I keep getting the error message: Parallel Read times: 0.0771379 PARALLEL READ PART 0.00908303 PARALLEL RESOLVE_SHARED_ENTS 0.000237942 PARALLEL RESOLVE_SHARED_SETS 0.0863409 PARALLEL TOTAL ABORT: nelv is invalid in nekmoab_proc_map ABORT: nelv is invalid in nekmoab_proc_map nelv, lelv = 0 64 ABORT: nelv is invalid in nekmoab_proc_map nelv, lelv = 0 64 nelv, lelv = 0 64 ABORT: nelv is invalid in nekmoab_proc_map nelv, lelv = 0 64 call exitt: dying ... This has been asked before and the problem was identified that moab enabled Nek5000 is not compatible with 2D quads. However I have a 3D mesh, proven by: type count total ------------------------------ Edge 282 3.7e+02 Quad 152 2.6e+02 Hex 64 1.2e+02 1D Side 1376 1.8e+03 Vertex 765 A very small mesh I know. So the issue seems to be that nelv is less than 64 and in the moab.f routine this brings the above error (I've tried simply commenting this out unsuccessfully). Thus it would seem logical that one defines nelv to be the correct value. I have tried this by defining it in the geometry section of the .rea file and this brings even more errors. To my knowledge as my mesh is 3D this subroutine shouldn't even be called. Any help is appreciated! Thanks for your time, Friedrich. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Mon Oct 20 02:03:12 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 20 Oct 2014 09:03:12 +0200 Subject: [Nek5000-users] Interfacing with Arpack Message-ID: Dear users and developers, I would like to solve a generalized eigenvalue problem with Arpack, based on Nek subroutines. Using reverse communication interface, I only need to express the action of the matrices on the vectors, but I am not sure on how to apply the mass matrix and the inverse mass matrix in the Pn-Pn formulation. Is it sufficient to write: call col2(p1,bm1,n) !apply mass matrix to vector p1 ... call invcol2(p1,bm1,n) ! apply inverse mass matrix to vector p1 or should I do: call rzero(h1,n) call rone(h2,n) call hmholtz('vel',p,p1,h1,h2,pmask,vmult,imsh,tolh,maxiter,isd) to apply inverse mass matrix to p1, or should I do something else? Also, from a related question from a couple of years ago, should I call col2(v,vmask,n) after each matrix-vector product? Thank you in advance. Best, Giuseppe From nek5000-users at lists.mcs.anl.gov Mon Oct 20 05:58:00 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 20 Oct 2014 10:58:00 +0000 Subject: [Nek5000-users] Interfacing with Arpack In-Reply-To: References: Message-ID: Giuseppe, The key thing to understand about Nek is that variables are _always_ stored in their local format, which implies you have redundant values at the interface such that they are ready for parallel operator evaluation. Let u_L denote such a vector with redundant values (L for Local) and let u be the corresponding vector without the redundantly stored data. Nominally, there is a matrix Q that maps u to u_L: (1) u_L = Q u and the matrix-vector product involving the mass matrix, of the form w = M u, can be expressed as: (2) w = M u = Q^T M_L Q u = Q^T M_L u_L where M_L = block_diag(M^e) , e=1,...,Nelv is the block-diagonal matrix consisting of local mass matrices within each element, e. In Nek, M^e is also block diagonal, so the whole _unassembled_ stiffness matrix, M_L, is diagonal. It is this matrix that is stored in bm1 ("B" on Mesh 1, the velocity/temp mesh). Notice that, as written, the output of (2) violates our "store only local vectors" policy, so we invoke (1) to correct this: (3) w_L = Q w = Q M u = Q Q^T M_L u_L. We refer to QQ^T as direct-stiffness summation (dssum, in nek). It is the operation that glues things together. So, suppose you wish to find a continuous function that is close to a discontinuous function in nek. Let f be the continuous thing and g be the discontinuous one. You can do this via: (4) f_L = Q M^-1 Q M_L g_L. In Nek, this would be written as: n = nx1*ny1*nz1*nelv call col3(f,g,bm1,n) call dssum(f,nx1,ny1,nz1) call col2 (f,binvm1,n) Here, binvm1 is effectively binvm1 = 1./ (QQT*bm1) , if we view bm1 and binvm1 as vectors, rather than diagonal matrices. Paul ________________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Monday, October 20, 2014 2:03 AM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] Interfacing with Arpack Dear users and developers, I would like to solve a generalized eigenvalue problem with Arpack, based on Nek subroutines. Using reverse communication interface, I only need to express the action of the matrices on the vectors, but I am not sure on how to apply the mass matrix and the inverse mass matrix in the Pn-Pn formulation. Is it sufficient to write: call col2(p1,bm1,n) !apply mass matrix to vector p1 ... call invcol2(p1,bm1,n) ! apply inverse mass matrix to vector p1 or should I do: call rzero(h1,n) call rone(h2,n) call hmholtz('vel',p,p1,h1,h2,pmask,vmult,imsh,tolh,maxiter,isd) to apply inverse mass matrix to p1, or should I do something else? Also, from a related question from a couple of years ago, should I call col2(v,vmask,n) after each matrix-vector product? Thank you in advance. Best, Giuseppe _______________________________________________ Nek5000-users mailing list Nek5000-users at lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Mon Oct 20 08:21:11 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 20 Oct 2014 15:21:11 +0200 Subject: [Nek5000-users] Interfacing with Arpack In-Reply-To: References: Message-ID: Dear Paul, thank you for the answer. If my understanding is correct, since each matrix-vector product (coded as matrix-matrix product) is done on the local variables, I should call dssum after each matrix-vector operation. For instance, if I want to apply the Helmholtz matrix to a vector u and store the result in v, I should do: call axhelm(v,u,h1,h2,imsh,1) call dssum(v,nx1,ny1,nz1) is it correct? If I want to apply the gradient to a vector (always for eigenvalue computation purposes) shoud I do something like: call gradm1(dudx,dudy,dudz,u) call dssum(dudx,nx1,ny1,nz1) call dssum(dudy,nx1,ny1,nz1) call dssum(dudz,nx1,ny1,nz1) bufx = glsc2(u0,dudx,n) !bufx = u0.du/dx or am I missing something? Giuseppe On 20/10/2014 12:58, nek5000-users at lists.mcs.anl.gov wrote: > Giuseppe, > > The key thing to understand about Nek is that variables are _always_ stored in their local format, > which implies you have redundant values at the interface such that they are ready for parallel operator > evaluation. > > Let u_L denote such a vector with redundant values (L for Local) and let u be the corresponding > vector without the redundantly stored data. Nominally, there is a matrix Q that maps u to u_L: > > (1) u_L = Q u > > and the matrix-vector product involving the mass matrix, of the form w = M u, can be expressed as: > > (2) w = M u = Q^T M_L Q u = Q^T M_L u_L > > where M_L = block_diag(M^e) , e=1,...,Nelv is the block-diagonal matrix consisting of local mass > matrices within each element, e. In Nek, M^e is also block diagonal, so the whole _unassembled_ > stiffness matrix, M_L, is diagonal. It is this matrix that is stored in bm1 ("B" on Mesh 1, the velocity/temp > mesh). > > Notice that, as written, the output of (2) violates our "store only local vectors" policy, so we invoke > (1) to correct this: > > > (3) w_L = Q w = Q M u = Q Q^T M_L u_L. > > > We refer to QQ^T as direct-stiffness summation (dssum, in nek). It is the operation that > glues things together. > > So, suppose you wish to find a continuous function that is close to a discontinuous function > in nek. Let f be the continuous thing and g be the discontinuous one. You can do this via: > > (4) f_L = Q M^-1 Q M_L g_L. > > In Nek, this would be written as: > > n = nx1*ny1*nz1*nelv > > call col3(f,g,bm1,n) > > call dssum(f,nx1,ny1,nz1) > > call col2 (f,binvm1,n) > > Here, binvm1 is effectively binvm1 = 1./ (QQT*bm1) , if we view bm1 and binvm1 as > vectors, rather than diagonal matrices. > > Paul > > > > > > ________________________________________ > From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > Sent: Monday, October 20, 2014 2:03 AM > To: nek5000-users at lists.mcs.anl.gov > Subject: [Nek5000-users] Interfacing with Arpack > > Dear users and developers, > I would like to solve a generalized eigenvalue problem with Arpack, > based on Nek subroutines. Using reverse communication interface, I only > need to express the action of the matrices on the vectors, but I am not > sure on how to apply the mass matrix and the inverse mass matrix in the > Pn-Pn formulation. > Is it sufficient to write: > call col2(p1,bm1,n) !apply mass matrix to vector p1 > ... > call invcol2(p1,bm1,n) ! apply inverse mass matrix to vector p1 > or should I do: > call rzero(h1,n) > call rone(h2,n) > call hmholtz('vel',p,p1,h1,h2,pmask,vmult,imsh,tolh,maxiter,isd) > to apply inverse mass matrix to p1, or should I do something else? > Also, from a related question from a couple of years ago, should I call > col2(v,vmask,n) after each matrix-vector product? > Thank you in advance. > Best, > Giuseppe > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Mon Oct 20 09:37:24 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 20 Oct 2014 14:37:24 +0000 Subject: [Nek5000-users] Interfacing with Arpack In-Reply-To: References: , Message-ID: Hi Giuseppe, You're close. The key point to understand is that application of Q^T arises from continuity requirements on the test functions (that we normally call "v" in an abstract setting). Everything in Nek is cast in the weak form: Find u \in X^N_0 such that, for all v \in X^N_0, a( v, u )_N = a( v, u* )_N where u* is the exact solution and a( . , . )_N is the inner-product of interest, with discrete quadrature. For example, if a( v , u ) := \int_Omega grad v . grad u dV (1) then a( v , u )_N would be the same bilinear form with integration replaced by numerical quadrature. Note that an L2 projection would be of the form: ( v , u ) = ( v , u* ) where I've dropped the _N on the inner product on the working assumption that we always use quadrature. Since u,v are continuous in Nek, the bilinear forms read: v^T Q^T M_L Q u = v^T Q^T M_L u*_L and, in nek, u = Binvm1 * QQ^T BM1 u* Note that (1), above, has the mass matrix build in because of the volume integral. So, technically, if you are doing dssum, you should have a mass matrix present, because dssum relates to projection against the test functions v, which implies a volume-weighted average of the equations. Sorry if this isn't very clear... Paul ________________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Monday, October 20, 2014 8:21 AM To: nek5000-users at lists.mcs.anl.gov Subject: Re: [Nek5000-users] Interfacing with Arpack Dear Paul, thank you for the answer. If my understanding is correct, since each matrix-vector product (coded as matrix-matrix product) is done on the local variables, I should call dssum after each matrix-vector operation. For instance, if I want to apply the Helmholtz matrix to a vector u and store the result in v, I should do: call axhelm(v,u,h1,h2,imsh,1) call dssum(v,nx1,ny1,nz1) is it correct? If I want to apply the gradient to a vector (always for eigenvalue computation purposes) shoud I do something like: call gradm1(dudx,dudy,dudz,u) call dssum(dudx,nx1,ny1,nz1) call dssum(dudy,nx1,ny1,nz1) call dssum(dudz,nx1,ny1,nz1) bufx = glsc2(u0,dudx,n) !bufx = u0.du/dx or am I missing something? Giuseppe On 20/10/2014 12:58, nek5000-users at lists.mcs.anl.gov wrote: > Giuseppe, > > The key thing to understand about Nek is that variables are _always_ stored in their local format, > which implies you have redundant values at the interface such that they are ready for parallel operator > evaluation. > > Let u_L denote such a vector with redundant values (L for Local) and let u be the corresponding > vector without the redundantly stored data. Nominally, there is a matrix Q that maps u to u_L: > > (1) u_L = Q u > > and the matrix-vector product involving the mass matrix, of the form w = M u, can be expressed as: > > (2) w = M u = Q^T M_L Q u = Q^T M_L u_L > > where M_L = block_diag(M^e) , e=1,...,Nelv is the block-diagonal matrix consisting of local mass > matrices within each element, e. In Nek, M^e is also block diagonal, so the whole _unassembled_ > stiffness matrix, M_L, is diagonal. It is this matrix that is stored in bm1 ("B" on Mesh 1, the velocity/temp > mesh). > > Notice that, as written, the output of (2) violates our "store only local vectors" policy, so we invoke > (1) to correct this: > > > (3) w_L = Q w = Q M u = Q Q^T M_L u_L. > > > We refer to QQ^T as direct-stiffness summation (dssum, in nek). It is the operation that > glues things together. > > So, suppose you wish to find a continuous function that is close to a discontinuous function > in nek. Let f be the continuous thing and g be the discontinuous one. You can do this via: > > (4) f_L = Q M^-1 Q M_L g_L. > > In Nek, this would be written as: > > n = nx1*ny1*nz1*nelv > > call col3(f,g,bm1,n) > > call dssum(f,nx1,ny1,nz1) > > call col2 (f,binvm1,n) > > Here, binvm1 is effectively binvm1 = 1./ (QQT*bm1) , if we view bm1 and binvm1 as > vectors, rather than diagonal matrices. > > Paul > > > > > > ________________________________________ > From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > Sent: Monday, October 20, 2014 2:03 AM > To: nek5000-users at lists.mcs.anl.gov > Subject: [Nek5000-users] Interfacing with Arpack > > Dear users and developers, > I would like to solve a generalized eigenvalue problem with Arpack, > based on Nek subroutines. Using reverse communication interface, I only > need to express the action of the matrices on the vectors, but I am not > sure on how to apply the mass matrix and the inverse mass matrix in the > Pn-Pn formulation. > Is it sufficient to write: > call col2(p1,bm1,n) !apply mass matrix to vector p1 > ... > call invcol2(p1,bm1,n) ! apply inverse mass matrix to vector p1 > or should I do: > call rzero(h1,n) > call rone(h2,n) > call hmholtz('vel',p,p1,h1,h2,pmask,vmult,imsh,tolh,maxiter,isd) > to apply inverse mass matrix to p1, or should I do something else? > Also, from a related question from a couple of years ago, should I call > col2(v,vmask,n) after each matrix-vector product? > Thank you in advance. > Best, > Giuseppe > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users _______________________________________________ Nek5000-users mailing list Nek5000-users at lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Tue Oct 21 10:23:48 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 21 Oct 2014 15:23:48 +0000 Subject: [Nek5000-users] Periodic Boundary Definition Message-ID: Hi All, When using nek/moab how are the periodic boundaries to be defined? Currently I have this: channel.h5m 1 0 1 fluid set, 0 other/solid sets 10 fluid set #10, solid sets #0 1 Material properties -fluid set #100: MATINDX of 1, All elements: IMATIE = 1 6 no. bc sets; bc set id, bc type: (f=flux, c=convective, t=dirichlet, I=adiabatic) 100 1 P , 200 1 P , 300 1 W , 400 1 W , 500 1 P , 600 1 P , Boundaries 100 and 200 need to be defined periodic with one another... I have experimented with: 100 1 P , 200 1 200 1 P , 100 1 and 100 1 200 1 P , 200 1 100 1 P , Neither of which works, is there another implementation or is periodic boundary not possible? Thanks, Friedrich. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Tue Oct 21 11:17:13 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 21 Oct 2014 16:17:13 +0000 Subject: [Nek5000-users] Periodic Boundary Definition In-Reply-To: References: Message-ID: Hi Friedrich, Nek-MOAB implementation does not support periodic BCs since it was envisioned to work on Cubit meshes but Cubit could not guarantee exactly the same element face layout for the two boundaries so the grid point-to-gridpoint periodicity was impossible to implement. Could you use an inlet and outlet BC instead? Aleks ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Tuesday, October 21, 2014 10:23 AM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] Periodic Boundary Definition Hi All, When using nek/moab how are the periodic boundaries to be defined? Currently I have this: channel.h5m 1 0 1 fluid set, 0 other/solid sets 10 fluid set #10, solid sets #0 1 Material properties -fluid set #100: MATINDX of 1, All elements: IMATIE = 1 6 no. bc sets; bc set id, bc type: (f=flux, c=convective, t=dirichlet, I=adiabatic) 100 1 P , 200 1 P , 300 1 W , 400 1 W , 500 1 P , 600 1 P , Boundaries 100 and 200 need to be defined periodic with one another... I have experimented with: 100 1 P , 200 1 200 1 P , 100 1 and 100 1 200 1 P , 200 1 100 1 P , Neither of which works, is there another implementation or is periodic boundary not possible? Thanks, Friedrich. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Tue Oct 21 23:18:06 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 22 Oct 2014 09:48:06 +0530 Subject: [Nek5000-users] Restarting simulation in same run Message-ID: Dear neks I want to perform several simulations each with different initial condition in the same run of nek5000. This is for eigenvalue/vector computation with ARPACK and using time-stepping approach. I plan to do all of this in userchk. The call to ARPACK will be in userchk. Whenever ARPACK gives me a vector, I want to start a simulation with this vector as initial condition. I have to perform certain number of time-steps and return the result to ARPACK. What all do I need to do to initialize a simulation given the initial condition vector ? Thanks praveen -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Wed Oct 22 01:53:57 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 22 Oct 2014 08:53:57 +0200 Subject: [Nek5000-users] Interfacing with Arpack In-Reply-To: References: , Message-ID: Hi Paul, thanks again for your reply. I think I am close to a solution now. One more thing: I think I need to mask out Dirichlet bc nodes when applying the matrix-vector products, right? How can I do this? Is it sufficient to call col2(u,vmask,n) after the matrix-vector product? Best, Giuseppe On 20/10/2014 16:37, nek5000-users at lists.mcs.anl.gov wrote: > Hi Giuseppe, > > You're close. > > The key point to understand is that application of Q^T arises from continuity > requirements on the test functions (that we normally call "v" in an abstract setting). > Everything in Nek is cast in the weak form: Find u \in X^N_0 such that, for all v \in X^N_0, > > a( v, u )_N = a( v, u* )_N > > where u* is the exact solution and a( . , . )_N is the inner-product of interest, with discrete > quadrature. For example, if > > a( v , u ) := \int_Omega grad v . grad u dV (1) > > then a( v , u )_N would be the same bilinear form with integration replaced by numerical > quadrature. Note that an L2 projection would be of the form: > > > ( v , u ) = ( v , u* ) > > where I've dropped the _N on the inner product on the working assumption that we > always use quadrature. > > Since u,v are continuous in Nek, the bilinear forms read: > > v^T Q^T M_L Q u = v^T Q^T M_L u*_L > > and, in nek, > > u = Binvm1 * QQ^T BM1 u* > > Note that (1), above, has the mass matrix build in because of the volume > integral. So, technically, if you are doing dssum, you should have a mass > matrix present, because dssum relates to projection against the test functions > v, which implies a volume-weighted average of the equations. > > Sorry if this isn't very clear... > > Paul > > > > > ________________________________________ > From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > Sent: Monday, October 20, 2014 8:21 AM > To: nek5000-users at lists.mcs.anl.gov > Subject: Re: [Nek5000-users] Interfacing with Arpack > > Dear Paul, > thank you for the answer. > If my understanding is correct, since each matrix-vector product (coded > as matrix-matrix product) is done on the local variables, I should call > dssum after each matrix-vector operation. > For instance, if I want to apply the Helmholtz matrix to a vector u and > store the result in v, I should do: > call axhelm(v,u,h1,h2,imsh,1) > call dssum(v,nx1,ny1,nz1) > is it correct? > If I want to apply the gradient to a vector (always for eigenvalue > computation purposes) shoud I do something like: > call gradm1(dudx,dudy,dudz,u) > call dssum(dudx,nx1,ny1,nz1) > call dssum(dudy,nx1,ny1,nz1) > call dssum(dudz,nx1,ny1,nz1) > bufx = glsc2(u0,dudx,n) !bufx = u0.du/dx > or am I missing something? > > Giuseppe > > > > On 20/10/2014 12:58, nek5000-users at lists.mcs.anl.gov wrote: >> Giuseppe, >> >> The key thing to understand about Nek is that variables are _always_ stored in their local format, >> which implies you have redundant values at the interface such that they are ready for parallel operator >> evaluation. >> >> Let u_L denote such a vector with redundant values (L for Local) and let u be the corresponding >> vector without the redundantly stored data. Nominally, there is a matrix Q that maps u to u_L: >> >> (1) u_L = Q u >> >> and the matrix-vector product involving the mass matrix, of the form w = M u, can be expressed as: >> >> (2) w = M u = Q^T M_L Q u = Q^T M_L u_L >> >> where M_L = block_diag(M^e) , e=1,...,Nelv is the block-diagonal matrix consisting of local mass >> matrices within each element, e. In Nek, M^e is also block diagonal, so the whole _unassembled_ >> stiffness matrix, M_L, is diagonal. It is this matrix that is stored in bm1 ("B" on Mesh 1, the velocity/temp >> mesh). >> >> Notice that, as written, the output of (2) violates our "store only local vectors" policy, so we invoke >> (1) to correct this: >> >> >> (3) w_L = Q w = Q M u = Q Q^T M_L u_L. >> >> >> We refer to QQ^T as direct-stiffness summation (dssum, in nek). It is the operation that >> glues things together. >> >> So, suppose you wish to find a continuous function that is close to a discontinuous function >> in nek. Let f be the continuous thing and g be the discontinuous one. You can do this via: >> >> (4) f_L = Q M^-1 Q M_L g_L. >> >> In Nek, this would be written as: >> >> n = nx1*ny1*nz1*nelv >> >> call col3(f,g,bm1,n) >> >> call dssum(f,nx1,ny1,nz1) >> >> call col2 (f,binvm1,n) >> >> Here, binvm1 is effectively binvm1 = 1./ (QQT*bm1) , if we view bm1 and binvm1 as >> vectors, rather than diagonal matrices. >> >> Paul >> >> >> >> >> >> ________________________________________ >> From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >> Sent: Monday, October 20, 2014 2:03 AM >> To: nek5000-users at lists.mcs.anl.gov >> Subject: [Nek5000-users] Interfacing with Arpack >> >> Dear users and developers, >> I would like to solve a generalized eigenvalue problem with Arpack, >> based on Nek subroutines. Using reverse communication interface, I only >> need to express the action of the matrices on the vectors, but I am not >> sure on how to apply the mass matrix and the inverse mass matrix in the >> Pn-Pn formulation. >> Is it sufficient to write: >> call col2(p1,bm1,n) !apply mass matrix to vector p1 >> ... >> call invcol2(p1,bm1,n) ! apply inverse mass matrix to vector p1 >> or should I do: >> call rzero(h1,n) >> call rone(h2,n) >> call hmholtz('vel',p,p1,h1,h2,pmask,vmult,imsh,tolh,maxiter,isd) >> to apply inverse mass matrix to p1, or should I do something else? >> Also, from a related question from a couple of years ago, should I call >> col2(v,vmask,n) after each matrix-vector product? >> Thank you in advance. >> Best, >> Giuseppe >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Wed Oct 22 06:31:08 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 22 Oct 2014 06:31:08 -0500 (CDT) Subject: [Nek5000-users] Interfacing with Arpack In-Reply-To: References: , Message-ID: Giuseppe, There are several masks: tmask() v1mask for vx v2mask for vy v3mask for vz pmask for pressure tmask has several fields in the situation where you have different boundary conditions on different passive scalars. Note that repeated application of one of the SEM operators requires injection into H^1, which is effected through dssum + mass matrix, as discussed earlier. Subroutine ALPHAM1 in eigsolv illustrates this. (It's an older routine, so it uses the multiplicity array when computing this inner products. That would not be necessary, however, if the inner products had been computed prior to dssum.) Paul On Wed, 22 Oct 2014, nek5000-users at lists.mcs.anl.gov wrote: > Hi Paul, thanks again for your reply. I think I am close to a solution now. > One more thing: I think I need to mask out Dirichlet bc nodes when applying > the matrix-vector products, right? How can I do this? Is it sufficient to > call col2(u,vmask,n) after the matrix-vector product? > Best, > Giuseppe > > > On 20/10/2014 16:37, nek5000-users at lists.mcs.anl.gov wrote: >> Hi Giuseppe, >> >> You're close. >> >> The key point to understand is that application of Q^T arises from >> continuity >> requirements on the test functions (that we normally call "v" in an >> abstract setting). >> Everything in Nek is cast in the weak form: Find u \in X^N_0 such that, >> for all v \in X^N_0, >> >> a( v, u )_N = a( v, u* )_N >> >> where u* is the exact solution and a( . , . )_N is the inner-product of >> interest, with discrete >> quadrature. For example, if >> >> a( v , u ) := \int_Omega grad v . grad u dV (1) >> >> then a( v , u )_N would be the same bilinear form with integration replaced >> by numerical >> quadrature. Note that an L2 projection would be of the form: >> >> >> ( v , u ) = ( v , u* ) >> >> where I've dropped the _N on the inner product on the working assumption >> that we >> always use quadrature. >> >> Since u,v are continuous in Nek, the bilinear forms read: >> >> v^T Q^T M_L Q u = v^T Q^T M_L u*_L >> >> and, in nek, >> >> u = Binvm1 * QQ^T BM1 u* >> >> Note that (1), above, has the mass matrix build in because of the volume >> integral. So, technically, if you are doing dssum, you should have a >> mass >> matrix present, because dssum relates to projection against the test >> functions >> v, which implies a volume-weighted average of the equations. >> >> Sorry if this isn't very clear... >> >> Paul >> >> >> >> >> ________________________________________ >> From: nek5000-users-bounces at lists.mcs.anl.gov >> [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of >> nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >> Sent: Monday, October 20, 2014 8:21 AM >> To: nek5000-users at lists.mcs.anl.gov >> Subject: Re: [Nek5000-users] Interfacing with Arpack >> >> Dear Paul, >> thank you for the answer. >> If my understanding is correct, since each matrix-vector product (coded >> as matrix-matrix product) is done on the local variables, I should call >> dssum after each matrix-vector operation. >> For instance, if I want to apply the Helmholtz matrix to a vector u and >> store the result in v, I should do: >> call axhelm(v,u,h1,h2,imsh,1) >> call dssum(v,nx1,ny1,nz1) >> is it correct? >> If I want to apply the gradient to a vector (always for eigenvalue >> computation purposes) shoud I do something like: >> call gradm1(dudx,dudy,dudz,u) >> call dssum(dudx,nx1,ny1,nz1) >> call dssum(dudy,nx1,ny1,nz1) >> call dssum(dudz,nx1,ny1,nz1) >> bufx = glsc2(u0,dudx,n) !bufx = u0.du/dx >> or am I missing something? >> >> Giuseppe >> >> >> >> On 20/10/2014 12:58, nek5000-users at lists.mcs.anl.gov wrote: >>> Giuseppe, >>> >>> The key thing to understand about Nek is that variables are _always_ >>> stored in their local format, >>> which implies you have redundant values at the interface such that they >>> are ready for parallel operator >>> evaluation. >>> >>> Let u_L denote such a vector with redundant values (L for Local) and let u >>> be the corresponding >>> vector without the redundantly stored data. Nominally, there is a matrix >>> Q that maps u to u_L: >>> >>> (1) u_L = Q u >>> >>> and the matrix-vector product involving the mass matrix, of the form w = M >>> u, can be expressed as: >>> >>> (2) w = M u = Q^T M_L Q u = Q^T M_L u_L >>> >>> where M_L = block_diag(M^e) , e=1,...,Nelv is the block-diagonal matrix >>> consisting of local mass >>> matrices within each element, e. In Nek, M^e is also block diagonal, so >>> the whole _unassembled_ >>> stiffness matrix, M_L, is diagonal. It is this matrix that is stored in >>> bm1 ("B" on Mesh 1, the velocity/temp >>> mesh). >>> >>> Notice that, as written, the output of (2) violates our "store only local >>> vectors" policy, so we invoke >>> (1) to correct this: >>> >>> >>> (3) w_L = Q w = Q M u = Q Q^T M_L u_L. >>> >>> >>> We refer to QQ^T as direct-stiffness summation (dssum, in nek). It is >>> the operation that >>> glues things together. >>> >>> So, suppose you wish to find a continuous function that is close to a >>> discontinuous function >>> in nek. Let f be the continuous thing and g be the discontinuous one. >>> You can do this via: >>> >>> (4) f_L = Q M^-1 Q M_L g_L. >>> >>> In Nek, this would be written as: >>> >>> n = nx1*ny1*nz1*nelv >>> >>> call col3(f,g,bm1,n) >>> >>> call dssum(f,nx1,ny1,nz1) >>> >>> call col2 (f,binvm1,n) >>> >>> Here, binvm1 is effectively binvm1 = 1./ (QQT*bm1) , if we view bm1 >>> and binvm1 as >>> vectors, rather than diagonal matrices. >>> >>> Paul >>> >>> >>> >>> >>> >>> ________________________________________ >>> From: nek5000-users-bounces at lists.mcs.anl.gov >>> [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of >>> nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >>> Sent: Monday, October 20, 2014 2:03 AM >>> To: nek5000-users at lists.mcs.anl.gov >>> Subject: [Nek5000-users] Interfacing with Arpack >>> >>> Dear users and developers, >>> I would like to solve a generalized eigenvalue problem with Arpack, >>> based on Nek subroutines. Using reverse communication interface, I only >>> need to express the action of the matrices on the vectors, but I am not >>> sure on how to apply the mass matrix and the inverse mass matrix in the >>> Pn-Pn formulation. >>> Is it sufficient to write: >>> call col2(p1,bm1,n) !apply mass matrix to vector p1 >>> ... >>> call invcol2(p1,bm1,n) ! apply inverse mass matrix to vector p1 >>> or should I do: >>> call rzero(h1,n) >>> call rone(h2,n) >>> call hmholtz('vel',p,p1,h1,h2,pmask,vmult,imsh,tolh,maxiter,isd) >>> to apply inverse mass matrix to p1, or should I do something else? >>> Also, from a related question from a couple of years ago, should I call >>> col2(v,vmask,n) after each matrix-vector product? >>> Thank you in advance. >>> Best, >>> Giuseppe >>> _______________________________________________ >>> Nek5000-users mailing list >>> Nek5000-users at lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>> _______________________________________________ >>> Nek5000-users mailing list >>> Nek5000-users at lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > From nek5000-users at lists.mcs.anl.gov Wed Oct 22 06:36:47 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 22 Oct 2014 06:36:47 -0500 (CDT) Subject: [Nek5000-users] Restarting simulation in same run In-Reply-To: References: Message-ID: Hi Praveen, I would guess you need to develop your own timestepper because Nek uses BDFk & EXTk to time march, where typically k=3. At start-up, it uses k=1 and k=2 to bootstrap. If you start timestepping in the middle of the simulation then you need to overwrite the existing coefficients so that you can restart the timestepper, plus you need to store your initial condition (which presumably amounts to overwriting vx,vy,vz, and pr). There are others in the community who have done these kinds of things (I have not), who might have some further comments. Best, Paul PS ----- Are you trying to find eigenvalues associated with the spatial operators ? i.e., does the accuracy of the timestepper matter? If not, you can likely get away with k=1, in which case you don't need much history in the backward difference formula or extrapolation. Pressure, however, is a bit of a tricky business because we use splitting and there is a residual O(dt^k) impact... On Tue, 21 Oct 2014, nek5000-users at lists.mcs.anl.gov wrote: > Dear neks > > I want to perform several simulations each with different initial condition > in the same run of nek5000. This is for eigenvalue/vector computation with > ARPACK and using time-stepping approach. > > I plan to do all of this in userchk. The call to ARPACK will be in userchk. > Whenever ARPACK gives me a vector, I want to start a simulation with this > vector as initial condition. I have to perform certain number of time-steps > and return the result to ARPACK. > > What all do I need to do to initialize a simulation given the initial > condition vector ? > > Thanks > praveen > From nek5000-users at lists.mcs.anl.gov Wed Oct 22 08:07:08 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 22 Oct 2014 18:37:08 +0530 Subject: [Nek5000-users] Restarting simulation in same run In-Reply-To: References: Message-ID: Hello Paul I dont want to restart in the middle. Every time I want to start the time stepping from scratch. So I want to know how to reset everything for a fresh simulation. Every time ARPACK gives me a vector, I have to start with BDF1->BDF2->BDF3->BDF3... until final time. I think accuracy of time stepper also matters since we are approximating exact evolution operator exp(A*t) with the time stepper. I have read some papers on time stepping approach but the details are not clear. Thanks praveen On Wed, Oct 22, 2014 at 5:06 PM, wrote: > Hi Praveen, > > I would guess you need to develop your own timestepper because > Nek uses BDFk & EXTk to time march, where typically k=3. At > start-up, it uses k=1 and k=2 to bootstrap. If you start timestepping > in the middle of the simulation then you need to overwrite the > existing coefficients so that you can restart the timestepper, > plus you need to store your initial condition (which presumably > amounts to overwriting vx,vy,vz, and pr). > > There are others in the community who have done these kinds of > things (I have not), who might have some further comments. > > Best, > > Paul > > PS ----- Are you trying to find eigenvalues associated with > the spatial operators ? i.e., does the accuracy of the > timestepper matter? If not, you can likely get away with > k=1, in which case you don't need much history in the backward > difference formula or extrapolation. Pressure, however, is > a bit of a tricky business because we use splitting and there > is a residual O(dt^k) impact... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Fri Oct 24 05:41:08 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Fri, 24 Oct 2014 16:11:08 +0530 Subject: [Nek5000-users] SYM and ON boundary conditions Message-ID: Dear neks Could you list the equations used for the SYM and ON boundary conditions ? Thanks praveen -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Fri Oct 24 09:30:52 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Fri, 24 Oct 2014 14:30:52 +0000 Subject: [Nek5000-users] SYM and ON boundary conditions In-Reply-To: References: Message-ID: Hi Praveen, This information will be in the new documentation soon to be posted. I attached a small pdf of the section that interests you. I'd greatly appreciate appreciate if you let me know whether this was accurate and helped cause I didn't get to check it thoroughly yet (that's why the document is not up yet). Best, Oana ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Friday, October 24, 2014 5:41 AM To: nek5000 Subject: [Nek5000-users] SYM and ON boundary conditions Dear neks Could you list the equations used for the SYM and ON boundary conditions ? Thanks praveen -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: header_test.pdf Type: application/pdf Size: 64186 bytes Desc: header_test.pdf URL: From nek5000-users at lists.mcs.anl.gov Fri Oct 24 10:18:55 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Fri, 24 Oct 2014 20:48:55 +0530 Subject: [Nek5000-users] SYM and ON boundary conditions In-Reply-To: References: Message-ID: Hello Oana The "O" condition is fine. Is ON also same ? In the SYM, is there no condition on the pressure ? Also it seems to me it should read (grad(u).n).t or in terms of matrix-vector products t^T * grad(u) * n Can you clarify please ? Thanks praveen On Fri, Oct 24, 2014 at 8:00 PM, wrote: > Hi Praveen, > > This information will be in the new documentation soon to be posted. I > attached a small pdf of the section that interests you. > > I'd greatly appreciate appreciate if you let me know whether this was > accurate and helped cause I didn't get to check it thoroughly yet (that's > why the document is not up yet). > > Best, > Oana > ------------------------------ > *From:* nek5000-users-bounces at lists.mcs.anl.gov [ > nek5000-users-bounces at lists.mcs.anl.gov] on behalf of > nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > *Sent:* Friday, October 24, 2014 5:41 AM > *To:* nek5000 > *Subject:* [Nek5000-users] SYM and ON boundary conditions > > Dear neks > > Could you list the equations used for the SYM and ON boundary conditions > ? > > Thanks > praveen > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Fri Oct 24 11:35:44 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Fri, 24 Oct 2014 16:35:44 +0000 Subject: [Nek5000-users] SYM and ON boundary conditions In-Reply-To: References: , Message-ID: Hi Praveen, I think 'ON' is the same as O for the n direction, and Dirichlet for the t directions. Concerning the SYM you are right it is (grad(u).n).t, it may be better to not have it in vector notation at all. However I don't think there is a condition on the pressure, but I guess one could check in the code. Oana ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Friday, October 24, 2014 10:18 AM To: nek5000-users at lists.mcs.anl.gov Subject: Re: [Nek5000-users] SYM and ON boundary conditions Hello Oana The "O" condition is fine. Is ON also same ? In the SYM, is there no condition on the pressure ? Also it seems to me it should read (grad(u).n).t or in terms of matrix-vector products t^T * grad(u) * n Can you clarify please ? Thanks praveen On Fri, Oct 24, 2014 at 8:00 PM, > wrote: Hi Praveen, This information will be in the new documentation soon to be posted. I attached a small pdf of the section that interests you. I'd greatly appreciate appreciate if you let me know whether this was accurate and helped cause I didn't get to check it thoroughly yet (that's why the document is not up yet). Best, Oana ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Friday, October 24, 2014 5:41 AM To: nek5000 Subject: [Nek5000-users] SYM and ON boundary conditions Dear neks Could you list the equations used for the SYM and ON boundary conditions ? Thanks praveen _______________________________________________ Nek5000-users mailing list Nek5000-users at lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Fri Oct 24 22:49:33 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sat, 25 Oct 2014 09:19:33 +0530 Subject: [Nek5000-users] SYM and ON boundary conditions In-Reply-To: References: Message-ID: Yes, the SYM conditions make sense. The pressure boundary terms vanish in the weak formulation as do the viscous boundary terms. I suppose u.n = 0 is imposed strongly while (grad(u).n).t = 0 is imposed weakly. These two together imply that grad(p).n = 0 (using the momentum eqn and zero divergence condition). Is the implementation of "ON" different from "O". Otherwise I dont see the need for two different types ? Thanks praveen On Fri, Oct 24, 2014 at 10:05 PM, wrote: > Hi Praveen, > > I think 'ON' is the same as O for the n direction, and Dirichlet for the > t > directions. > > Concerning the SYM you are right it is (grad(u).n).t, it may be better to > not have it in vector notation at all. However I don't think there is a > condition on the pressure, but I guess one could check in the code. > > Oana > ------------------------------ > *From:* nek5000-users-bounces at lists.mcs.anl.gov [ > nek5000-users-bounces at lists.mcs.anl.gov] on behalf of > nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > *Sent:* Friday, October 24, 2014 10:18 AM > *To:* nek5000-users at lists.mcs.anl.gov > *Subject:* Re: [Nek5000-users] SYM and ON boundary conditions > > Hello Oana > > The "O" condition is fine. Is ON also same ? > > In the SYM, is there no condition on the pressure ? Also it seems to me > it should read > > (grad(u).n).t > > or in terms of matrix-vector products > > t^T * grad(u) * n > > Can you clarify please ? > > Thanks > praveen > > On Fri, Oct 24, 2014 at 8:00 PM, wrote: > >> Hi Praveen, >> >> This information will be in the new documentation soon to be posted. I >> attached a small pdf of the section that interests you. >> >> I'd greatly appreciate appreciate if you let me know whether this was >> accurate and helped cause I didn't get to check it thoroughly yet (that's >> why the document is not up yet). >> >> Best, >> Oana >> ------------------------------ >> *From:* nek5000-users-bounces at lists.mcs.anl.gov [ >> nek5000-users-bounces at lists.mcs.anl.gov] on behalf of >> nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >> *Sent:* Friday, October 24, 2014 5:41 AM >> *To:* nek5000 >> *Subject:* [Nek5000-users] SYM and ON boundary conditions >> >> Dear neks >> >> Could you list the equations used for the SYM and ON boundary >> conditions ? >> >> Thanks >> praveen >> >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> >> > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Sat Oct 25 01:52:37 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sat, 25 Oct 2014 14:52:37 +0800 (GMT+08:00) Subject: [Nek5000-users] How to use genbox to generate a mesh for half 2D circle/ 3d sphere? Message-ID: Dear Nek users, I am working on generating a mesh for half circle (or 3d hemisphere). I want to know how to make the inner part of the mesh is a box. Then I can map that boundary to another box mesh together. Thanks you first. Best regards, Zhenyu -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Sat Oct 25 03:01:01 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sat, 25 Oct 2014 16:01:01 +0800 (GMT+08:00) Subject: [Nek5000-users] How to use genbox to generate a mesh for half 2D circle/ 3d sphere? In-Reply-To: References: Message-ID: More exactly, how to generate the half circle mesh with the so called "picture frame" type topology? Is it possible to do this work by hand made scripts and then feed it into genbox? Cheers, Zhenyu -----????----- ???: nek5000-users at lists.mcs.anl.gov ????: 2014-10-25 14:52:37 (???) ???: nek5000-users at lists.mcs.anl.gov ??: ??: [Nek5000-users] How to use genbox to generate a mesh for half 2D circle/ 3d sphere? Dear Nek users, I am working on generating a mesh for half circle (or 3d hemisphere). I want to know how to make the inner part of the mesh is a box. Then I can map that boundary to another box mesh together. Thanks you first. Best regards, Zhenyu -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Sat Oct 25 08:58:15 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sat, 25 Oct 2014 15:58:15 +0200 Subject: [Nek5000-users] outpost parallel error Message-ID: Dear Neks, in the usr file, after doing some manipulations on a known solution (loaded from file with load_fld(...)), I save the computed values with a call to outpost, like: call outpost2(u1(1,1,1,1),v1(1,1,1,1), & w1(1,1,1,1),p1(1,1,1,1), & p1(1,1,1,1),1,'arp') While this is working correctly with 1 rank, in parallel it fails, returning only the error message: schfile:/.../myrun.sch then it hangs for a while, with all processes consuming all the cpu time until they are killed (so the .sch file is empty). Does anybody have an idea on how this could be solved? Best, Giuseppe From nek5000-users at lists.mcs.anl.gov Sat Oct 25 13:39:30 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sat, 25 Oct 2014 18:39:30 +0000 Subject: [Nek5000-users] outpost parallel error In-Reply-To: References: Message-ID: Hi Giuseppe, It looks as though you didn't remove the .sch file prior to the start of your run? Nek is designed to not clobber the .sch file because they can contain information that might have resulted from very long runs. The onus is thus on the user to manage those files. The usual nek scripts move those files to blah.sch1 if there is already a blah.sch file. It's possible, however, that something is messed up in the logic of outpost2(). That rarely gets exercised. Is there a reason to not use outpost()? Paul ________________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Saturday, October 25, 2014 8:58 AM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] outpost parallel error Dear Neks, in the usr file, after doing some manipulations on a known solution (loaded from file with load_fld(...)), I save the computed values with a call to outpost, like: call outpost2(u1(1,1,1,1),v1(1,1,1,1), & w1(1,1,1,1),p1(1,1,1,1), & p1(1,1,1,1),1,'arp') While this is working correctly with 1 rank, in parallel it fails, returning only the error message: schfile:/.../myrun.sch then it hangs for a while, with all processes consuming all the cpu time until they are killed (so the .sch file is empty). Does anybody have an idea on how this could be solved? Best, Giuseppe _______________________________________________ Nek5000-users mailing list Nek5000-users at lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Sun Oct 26 02:36:34 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sun, 26 Oct 2014 08:36:34 +0100 Subject: [Nek5000-users] outpost parallel error In-Reply-To: References: Message-ID: Hi Paul, thank for the answer. I tried both removing the .sch file before each run and using outpost instead of outpost2. Unfortunately none of these solved my problem. I have checked that outpost works in parallel before the Arpack pdneupd subroutine is called, so my guess is that the Arpack subroutine is modifying some flags forcing outpost to write the .sch file, only when called by 2 or more ranks. If somebody has further comments, please let me know. Best, Giuseppe On 25/10/2014 20:39, nek5000-users at lists.mcs.anl.gov wrote: > Hi Giuseppe, > > It looks as though you didn't remove the .sch file prior to the start > of your run? > > Nek is designed to not clobber the .sch file because they can contain > information that might have resulted from very long runs. The onus > is thus on the user to manage those files. The usual nek scripts move > those files to blah.sch1 if there is already a blah.sch file. > > It's possible, however, that something is messed up in the logic of outpost2(). > That rarely gets exercised. Is there a reason to not use outpost()? > > Paul > > ________________________________________ > From:nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf ofnek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > Sent: Saturday, October 25, 2014 8:58 AM > To:nek5000-users at lists.mcs.anl.gov > Subject: [Nek5000-users] outpost parallel error > > Dear Neks, > in the usr file, after doing some manipulations on a known solution > (loaded from file with load_fld(...)), I save the computed values with a > call to outpost, like: > call outpost2(u1(1,1,1,1),v1(1,1,1,1), > & w1(1,1,1,1),p1(1,1,1,1), > & p1(1,1,1,1),1,'arp') > While this is working correctly with 1 rank, in parallel it fails, > returning only the error message: > schfile:/.../myrun.sch > then it hangs for a while, with all processes consuming all the cpu time > until they are killed (so the .sch file is empty). > Does anybody have an idea on how this could be solved? > Best, > Giuseppe > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Sun Oct 26 08:36:45 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sun, 26 Oct 2014 13:36:45 +0000 Subject: [Nek5000-users] outpost parallel error In-Reply-To: References: , Message-ID: If you comment out the arpack call, does it work? ________________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Sunday, October 26, 2014 2:36 AM To: nek5000-users at lists.mcs.anl.gov Subject: Re: [Nek5000-users] outpost parallel error Hi Paul, thank for the answer. I tried both removing the .sch file before each run and using outpost instead of outpost2. Unfortunately none of these solved my problem. I have checked that outpost works in parallel before the Arpack pdneupd subroutine is called, so my guess is that the Arpack subroutine is modifying some flags forcing outpost to write the .sch file, only when called by 2 or more ranks. If somebody has further comments, please let me know. Best, Giuseppe On 25/10/2014 20:39, nek5000-users at lists.mcs.anl.gov wrote: > Hi Giuseppe, > > It looks as though you didn't remove the .sch file prior to the start > of your run? > > Nek is designed to not clobber the .sch file because they can contain > information that might have resulted from very long runs. The onus > is thus on the user to manage those files. The usual nek scripts move > those files to blah.sch1 if there is already a blah.sch file. > > It's possible, however, that something is messed up in the logic of outpost2(). > That rarely gets exercised. Is there a reason to not use outpost()? > > Paul > > ________________________________________ > From:nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf ofnek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > Sent: Saturday, October 25, 2014 8:58 AM > To:nek5000-users at lists.mcs.anl.gov > Subject: [Nek5000-users] outpost parallel error > > Dear Neks, > in the usr file, after doing some manipulations on a known solution > (loaded from file with load_fld(...)), I save the computed values with a > call to outpost, like: > call outpost2(u1(1,1,1,1),v1(1,1,1,1), > & w1(1,1,1,1),p1(1,1,1,1), > & p1(1,1,1,1),1,'arp') > While this is working correctly with 1 rank, in parallel it fails, > returning only the error message: > schfile:/.../myrun.sch > then it hangs for a while, with all processes consuming all the cpu time > until they are killed (so the .sch file is empty). > Does anybody have an idea on how this could be solved? > Best, > Giuseppe > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users _______________________________________________ Nek5000-users mailing list Nek5000-users at lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Sun Oct 26 09:39:26 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sun, 26 Oct 2014 15:39:26 +0100 Subject: [Nek5000-users] outpost parallel error In-Reply-To: References: , Message-ID: My userchk is structured this way: ... call pdnaupd(...) ... c here call to nek subroutines c for matrix-vector products ... c if outpost is called here, it works (serial and parallel) c call to Arpack postprocessing routine: call pdneupd(...,zz,...) c zz is a matrix storing the eigenvectors c call to arpack to nek format conversion, then call outpost(v1,v2,...) ! this gets stuck in parallel note that with 1 rank this code is working well. If I comment out the call to pdneupd, then it works in parallel, although it will not give any useful results. Giuseppe On 26/10/2014 14:36, nek5000-users at lists.mcs.anl.gov wrote: > If you comment out the arpack call, does it work? > > ________________________________________ > From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > Sent: Sunday, October 26, 2014 2:36 AM > To: nek5000-users at lists.mcs.anl.gov > Subject: Re: [Nek5000-users] outpost parallel error > > Hi Paul, > thank for the answer. > I tried both removing the .sch file before each run and using outpost > instead of outpost2. > Unfortunately none of these solved my problem. > I have checked that outpost works in parallel before the Arpack pdneupd > subroutine is called, so my guess is that the Arpack subroutine is > modifying some flags forcing outpost to write the .sch file, only when > called by 2 or more ranks. > If somebody has further comments, please let me know. > Best, > Giuseppe > > > > On 25/10/2014 20:39, nek5000-users at lists.mcs.anl.gov wrote: >> Hi Giuseppe, >> >> It looks as though you didn't remove the .sch file prior to the start >> of your run? >> >> Nek is designed to not clobber the .sch file because they can contain >> information that might have resulted from very long runs. The onus >> is thus on the user to manage those files. The usual nek scripts move >> those files to blah.sch1 if there is already a blah.sch file. >> >> It's possible, however, that something is messed up in the logic of outpost2(). >> That rarely gets exercised. Is there a reason to not use outpost()? >> >> Paul >> >> ________________________________________ >> From:nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf ofnek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >> Sent: Saturday, October 25, 2014 8:58 AM >> To:nek5000-users at lists.mcs.anl.gov >> Subject: [Nek5000-users] outpost parallel error >> >> Dear Neks, >> in the usr file, after doing some manipulations on a known solution >> (loaded from file with load_fld(...)), I save the computed values with a >> call to outpost, like: >> call outpost2(u1(1,1,1,1),v1(1,1,1,1), >> & w1(1,1,1,1),p1(1,1,1,1), >> & p1(1,1,1,1),1,'arp') >> While this is working correctly with 1 rank, in parallel it fails, >> returning only the error message: >> schfile:/.../myrun.sch >> then it hangs for a while, with all processes consuming all the cpu time >> until they are killed (so the .sch file is empty). >> Does anybody have an idea on how this could be solved? >> Best, >> Giuseppe >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Sun Oct 26 11:30:06 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sun, 26 Oct 2014 16:30:06 +0000 Subject: [Nek5000-users] outpost parallel error In-Reply-To: References: , , Message-ID: Are all your arrays properly declared in the calling routine? Just trying to identify the potential issues... Best, Paul ________________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Sunday, October 26, 2014 9:39 AM To: nek5000-users at lists.mcs.anl.gov Subject: Re: [Nek5000-users] outpost parallel error My userchk is structured this way: ... call pdnaupd(...) ... c here call to nek subroutines c for matrix-vector products ... c if outpost is called here, it works (serial and parallel) c call to Arpack postprocessing routine: call pdneupd(...,zz,...) c zz is a matrix storing the eigenvectors c call to arpack to nek format conversion, then call outpost(v1,v2,...) ! this gets stuck in parallel note that with 1 rank this code is working well. If I comment out the call to pdneupd, then it works in parallel, although it will not give any useful results. Giuseppe On 26/10/2014 14:36, nek5000-users at lists.mcs.anl.gov wrote: > If you comment out the arpack call, does it work? > > ________________________________________ > From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > Sent: Sunday, October 26, 2014 2:36 AM > To: nek5000-users at lists.mcs.anl.gov > Subject: Re: [Nek5000-users] outpost parallel error > > Hi Paul, > thank for the answer. > I tried both removing the .sch file before each run and using outpost > instead of outpost2. > Unfortunately none of these solved my problem. > I have checked that outpost works in parallel before the Arpack pdneupd > subroutine is called, so my guess is that the Arpack subroutine is > modifying some flags forcing outpost to write the .sch file, only when > called by 2 or more ranks. > If somebody has further comments, please let me know. > Best, > Giuseppe > > > > On 25/10/2014 20:39, nek5000-users at lists.mcs.anl.gov wrote: >> Hi Giuseppe, >> >> It looks as though you didn't remove the .sch file prior to the start >> of your run? >> >> Nek is designed to not clobber the .sch file because they can contain >> information that might have resulted from very long runs. The onus >> is thus on the user to manage those files. The usual nek scripts move >> those files to blah.sch1 if there is already a blah.sch file. >> >> It's possible, however, that something is messed up in the logic of outpost2(). >> That rarely gets exercised. Is there a reason to not use outpost()? >> >> Paul >> >> ________________________________________ >> From:nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf ofnek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >> Sent: Saturday, October 25, 2014 8:58 AM >> To:nek5000-users at lists.mcs.anl.gov >> Subject: [Nek5000-users] outpost parallel error >> >> Dear Neks, >> in the usr file, after doing some manipulations on a known solution >> (loaded from file with load_fld(...)), I save the computed values with a >> call to outpost, like: >> call outpost2(u1(1,1,1,1),v1(1,1,1,1), >> & w1(1,1,1,1),p1(1,1,1,1), >> & p1(1,1,1,1),1,'arp') >> While this is working correctly with 1 rank, in parallel it fails, >> returning only the error message: >> schfile:/.../myrun.sch >> then it hangs for a while, with all processes consuming all the cpu time >> until they are killed (so the .sch file is empty). >> Does anybody have an idea on how this could be solved? >> Best, >> Giuseppe >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users _______________________________________________ Nek5000-users mailing list Nek5000-users at lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Sun Oct 26 12:20:00 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sun, 26 Oct 2014 18:20:00 +0100 Subject: [Nek5000-users] outpost parallel error In-Reply-To: References: , , Message-ID: Hi Paul, yes, all the arrays are declared. However, I have declared some arrays with the same name of some common declarations. For instance, I declare real*8 arrays named u,v,u0,v0. Could this be in conflict with some common /scruz/ declarations found in other source files? Thank you, Giuseppe On 26/10/2014 17:30, nek5000-users at lists.mcs.anl.gov wrote: > Are all your arrays properly declared in the calling routine? > > Just trying to identify the potential issues... > > Best, > > Paul > > ________________________________________ > From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > Sent: Sunday, October 26, 2014 9:39 AM > To: nek5000-users at lists.mcs.anl.gov > Subject: Re: [Nek5000-users] outpost parallel error > > My userchk is structured this way: > > ... > call pdnaupd(...) > ... > c here call to nek subroutines > c for matrix-vector products > ... > c if outpost is called here, it works (serial and parallel) > c call to Arpack postprocessing routine: > call pdneupd(...,zz,...) > c zz is a matrix storing the eigenvectors > c call to arpack to nek format conversion, then > call outpost(v1,v2,...) ! this gets stuck in parallel > > note that with 1 rank this code is working well. If I comment out the > call to pdneupd, then it works in parallel, although it will not give > any useful results. > Giuseppe > > > On 26/10/2014 14:36, nek5000-users at lists.mcs.anl.gov wrote: >> If you comment out the arpack call, does it work? >> >> ________________________________________ >> From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >> Sent: Sunday, October 26, 2014 2:36 AM >> To: nek5000-users at lists.mcs.anl.gov >> Subject: Re: [Nek5000-users] outpost parallel error >> >> Hi Paul, >> thank for the answer. >> I tried both removing the .sch file before each run and using outpost >> instead of outpost2. >> Unfortunately none of these solved my problem. >> I have checked that outpost works in parallel before the Arpack pdneupd >> subroutine is called, so my guess is that the Arpack subroutine is >> modifying some flags forcing outpost to write the .sch file, only when >> called by 2 or more ranks. >> If somebody has further comments, please let me know. >> Best, >> Giuseppe >> >> >> >> On 25/10/2014 20:39, nek5000-users at lists.mcs.anl.gov wrote: >>> Hi Giuseppe, >>> >>> It looks as though you didn't remove the .sch file prior to the start >>> of your run? >>> >>> Nek is designed to not clobber the .sch file because they can contain >>> information that might have resulted from very long runs. The onus >>> is thus on the user to manage those files. The usual nek scripts move >>> those files to blah.sch1 if there is already a blah.sch file. >>> >>> It's possible, however, that something is messed up in the logic of outpost2(). >>> That rarely gets exercised. Is there a reason to not use outpost()? >>> >>> Paul >>> >>> ________________________________________ >>> From:nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf ofnek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >>> Sent: Saturday, October 25, 2014 8:58 AM >>> To:nek5000-users at lists.mcs.anl.gov >>> Subject: [Nek5000-users] outpost parallel error >>> >>> Dear Neks, >>> in the usr file, after doing some manipulations on a known solution >>> (loaded from file with load_fld(...)), I save the computed values with a >>> call to outpost, like: >>> call outpost2(u1(1,1,1,1),v1(1,1,1,1), >>> & w1(1,1,1,1),p1(1,1,1,1), >>> & p1(1,1,1,1),1,'arp') >>> While this is working correctly with 1 rank, in parallel it fails, >>> returning only the error message: >>> schfile:/.../myrun.sch >>> then it hangs for a while, with all processes consuming all the cpu time >>> until they are killed (so the .sch file is empty). >>> Does anybody have an idea on how this could be solved? >>> Best, >>> Giuseppe >>> _______________________________________________ >>> Nek5000-users mailing list >>> Nek5000-users at lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>> _______________________________________________ >>> Nek5000-users mailing list >>> Nek5000-users at lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Sun Oct 26 12:40:14 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sun, 26 Oct 2014 17:40:14 +0000 Subject: [Nek5000-users] outpost parallel error In-Reply-To: References: , , , Message-ID: Hi Giuseppe, That shouldn't be an issue... Paul ________________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Sunday, October 26, 2014 12:20 PM To: nek5000-users at lists.mcs.anl.gov Subject: Re: [Nek5000-users] outpost parallel error Hi Paul, yes, all the arrays are declared. However, I have declared some arrays with the same name of some common declarations. For instance, I declare real*8 arrays named u,v,u0,v0. Could this be in conflict with some common /scruz/ declarations found in other source files? Thank you, Giuseppe On 26/10/2014 17:30, nek5000-users at lists.mcs.anl.gov wrote: > Are all your arrays properly declared in the calling routine? > > Just trying to identify the potential issues... > > Best, > > Paul > > ________________________________________ > From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > Sent: Sunday, October 26, 2014 9:39 AM > To: nek5000-users at lists.mcs.anl.gov > Subject: Re: [Nek5000-users] outpost parallel error > > My userchk is structured this way: > > ... > call pdnaupd(...) > ... > c here call to nek subroutines > c for matrix-vector products > ... > c if outpost is called here, it works (serial and parallel) > c call to Arpack postprocessing routine: > call pdneupd(...,zz,...) > c zz is a matrix storing the eigenvectors > c call to arpack to nek format conversion, then > call outpost(v1,v2,...) ! this gets stuck in parallel > > note that with 1 rank this code is working well. If I comment out the > call to pdneupd, then it works in parallel, although it will not give > any useful results. > Giuseppe > > > On 26/10/2014 14:36, nek5000-users at lists.mcs.anl.gov wrote: >> If you comment out the arpack call, does it work? >> >> ________________________________________ >> From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >> Sent: Sunday, October 26, 2014 2:36 AM >> To: nek5000-users at lists.mcs.anl.gov >> Subject: Re: [Nek5000-users] outpost parallel error >> >> Hi Paul, >> thank for the answer. >> I tried both removing the .sch file before each run and using outpost >> instead of outpost2. >> Unfortunately none of these solved my problem. >> I have checked that outpost works in parallel before the Arpack pdneupd >> subroutine is called, so my guess is that the Arpack subroutine is >> modifying some flags forcing outpost to write the .sch file, only when >> called by 2 or more ranks. >> If somebody has further comments, please let me know. >> Best, >> Giuseppe >> >> >> >> On 25/10/2014 20:39, nek5000-users at lists.mcs.anl.gov wrote: >>> Hi Giuseppe, >>> >>> It looks as though you didn't remove the .sch file prior to the start >>> of your run? >>> >>> Nek is designed to not clobber the .sch file because they can contain >>> information that might have resulted from very long runs. The onus >>> is thus on the user to manage those files. The usual nek scripts move >>> those files to blah.sch1 if there is already a blah.sch file. >>> >>> It's possible, however, that something is messed up in the logic of outpost2(). >>> That rarely gets exercised. Is there a reason to not use outpost()? >>> >>> Paul >>> >>> ________________________________________ >>> From:nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf ofnek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >>> Sent: Saturday, October 25, 2014 8:58 AM >>> To:nek5000-users at lists.mcs.anl.gov >>> Subject: [Nek5000-users] outpost parallel error >>> >>> Dear Neks, >>> in the usr file, after doing some manipulations on a known solution >>> (loaded from file with load_fld(...)), I save the computed values with a >>> call to outpost, like: >>> call outpost2(u1(1,1,1,1),v1(1,1,1,1), >>> & w1(1,1,1,1),p1(1,1,1,1), >>> & p1(1,1,1,1),1,'arp') >>> While this is working correctly with 1 rank, in parallel it fails, >>> returning only the error message: >>> schfile:/.../myrun.sch >>> then it hangs for a while, with all processes consuming all the cpu time >>> until they are killed (so the .sch file is empty). >>> Does anybody have an idea on how this could be solved? >>> Best, >>> Giuseppe >>> _______________________________________________ >>> Nek5000-users mailing list >>> Nek5000-users at lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>> _______________________________________________ >>> Nek5000-users mailing list >>> Nek5000-users at lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users _______________________________________________ Nek5000-users mailing list Nek5000-users at lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Sun Oct 26 14:10:15 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sun, 26 Oct 2014 20:10:15 +0100 Subject: [Nek5000-users] outpost parallel error In-Reply-To: References: , , , Message-ID: Hi Paul, I have solved the problem, there was indeed an error with mpi, but it was my fault, not Arpack's. Sorry for the inconvenience, and thanks for your help. Best, Giuseppe On 26/10/2014 18:40, nek5000-users at lists.mcs.anl.gov wrote: > Hi Giuseppe, > > That shouldn't be an issue... > > Paul > > ________________________________________ > From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] > Sent: Sunday, October 26, 2014 12:20 PM > To: nek5000-users at lists.mcs.anl.gov > Subject: Re: [Nek5000-users] outpost parallel error > > Hi Paul, > yes, all the arrays are declared. However, I have declared some arrays > with the same name of some common declarations. For instance, I declare > real*8 arrays named u,v,u0,v0. Could this be in conflict with some > common /scruz/ declarations found in other source files? > Thank you, > Giuseppe > > > > On 26/10/2014 17:30, nek5000-users at lists.mcs.anl.gov wrote: >> Are all your arrays properly declared in the calling routine? >> >> Just trying to identify the potential issues... >> >> Best, >> >> Paul >> >> ________________________________________ >> From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >> Sent: Sunday, October 26, 2014 9:39 AM >> To: nek5000-users at lists.mcs.anl.gov >> Subject: Re: [Nek5000-users] outpost parallel error >> >> My userchk is structured this way: >> >> ... >> call pdnaupd(...) >> ... >> c here call to nek subroutines >> c for matrix-vector products >> ... >> c if outpost is called here, it works (serial and parallel) >> c call to Arpack postprocessing routine: >> call pdneupd(...,zz,...) >> c zz is a matrix storing the eigenvectors >> c call to arpack to nek format conversion, then >> call outpost(v1,v2,...) ! this gets stuck in parallel >> >> note that with 1 rank this code is working well. If I comment out the >> call to pdneupd, then it works in parallel, although it will not give >> any useful results. >> Giuseppe >> >> >> On 26/10/2014 14:36, nek5000-users at lists.mcs.anl.gov wrote: >>> If you comment out the arpack call, does it work? >>> >>> ________________________________________ >>> From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >>> Sent: Sunday, October 26, 2014 2:36 AM >>> To: nek5000-users at lists.mcs.anl.gov >>> Subject: Re: [Nek5000-users] outpost parallel error >>> >>> Hi Paul, >>> thank for the answer. >>> I tried both removing the .sch file before each run and using outpost >>> instead of outpost2. >>> Unfortunately none of these solved my problem. >>> I have checked that outpost works in parallel before the Arpack pdneupd >>> subroutine is called, so my guess is that the Arpack subroutine is >>> modifying some flags forcing outpost to write the .sch file, only when >>> called by 2 or more ranks. >>> If somebody has further comments, please let me know. >>> Best, >>> Giuseppe >>> >>> >>> >>> On 25/10/2014 20:39, nek5000-users at lists.mcs.anl.gov wrote: >>>> Hi Giuseppe, >>>> >>>> It looks as though you didn't remove the .sch file prior to the start >>>> of your run? >>>> >>>> Nek is designed to not clobber the .sch file because they can contain >>>> information that might have resulted from very long runs. The onus >>>> is thus on the user to manage those files. The usual nek scripts move >>>> those files to blah.sch1 if there is already a blah.sch file. >>>> >>>> It's possible, however, that something is messed up in the logic of outpost2(). >>>> That rarely gets exercised. Is there a reason to not use outpost()? >>>> >>>> Paul >>>> >>>> ________________________________________ >>>> From:nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf ofnek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >>>> Sent: Saturday, October 25, 2014 8:58 AM >>>> To:nek5000-users at lists.mcs.anl.gov >>>> Subject: [Nek5000-users] outpost parallel error >>>> >>>> Dear Neks, >>>> in the usr file, after doing some manipulations on a known solution >>>> (loaded from file with load_fld(...)), I save the computed values with a >>>> call to outpost, like: >>>> call outpost2(u1(1,1,1,1),v1(1,1,1,1), >>>> & w1(1,1,1,1),p1(1,1,1,1), >>>> & p1(1,1,1,1),1,'arp') >>>> While this is working correctly with 1 rank, in parallel it fails, >>>> returning only the error message: >>>> schfile:/.../myrun.sch >>>> then it hangs for a while, with all processes consuming all the cpu time >>>> until they are killed (so the .sch file is empty). >>>> Does anybody have an idea on how this could be solved? >>>> Best, >>>> Giuseppe >>>> _______________________________________________ >>>> Nek5000-users mailing list >>>> Nek5000-users at lists.mcs.anl.gov >>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>>> _______________________________________________ >>>> Nek5000-users mailing list >>>> Nek5000-users at lists.mcs.anl.gov >>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>> _______________________________________________ >>> Nek5000-users mailing list >>> Nek5000-users at lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>> _______________________________________________ >>> Nek5000-users mailing list >>> Nek5000-users at lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Sun Oct 26 16:10:35 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Sun, 26 Oct 2014 17:10:35 -0400 Subject: [Nek5000-users] Curved GLL Points on Airfoil Message-ID: Hello Neks, I am trying to simulate the flow around an interpolated airfoil with curved boundaries. To do so I am generating the collocation points separately and writing them into an input file, which is read in the usrdat2 subroutine before geometry re-initialization. The only curved elements in the mesh are located at the airfoil boundary. I am not getting any errors during geometry re-initialization. However, the calculation starts off with a large courant number (around 7), which ends up diverging after 5 times steps. Along the element edges near the airfoil, there seems to be discontinuities in the flow which cause the solution to diverge. However, when inputting non-curvilinear collocation points, written in the exact same order to the input file, the simulation runs flawlessly. Are there any switches that must be activated in the rea file for curved elements? If not, are there any general guidelines to using curved elements in this manner? Thanks, Jeremie -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Sun Oct 26 20:38:53 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 27 Oct 2014 01:38:53 +0000 Subject: [Nek5000-users] Curved GLL Points on Airfoil In-Reply-To: References: Message-ID: Hi Jeremie, Try calling "fix_geom" in usrdat2 after you modify the sides... this might provide the requisite mesh clean up. Paul ________________________________ From: nek5000-users-bounces at lists.mcs.anl.gov [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] Sent: Sunday, October 26, 2014 4:10 PM To: nek5000-users at lists.mcs.anl.gov Subject: [Nek5000-users] Curved GLL Points on Airfoil Hello Neks, I am trying to simulate the flow around an interpolated airfoil with curved boundaries. To do so I am generating the collocation points separately and writing them into an input file, which is read in the usrdat2 subroutine before geometry re-initialization. The only curved elements in the mesh are located at the airfoil boundary. I am not getting any errors during geometry re-initialization. However, the calculation starts off with a large courant number (around 7), which ends up diverging after 5 times steps. Along the element edges near the airfoil, there seems to be discontinuities in the flow which cause the solution to diverge. However, when inputting non-curvilinear collocation points, written in the exact same order to the input file, the simulation runs flawlessly. Are there any switches that must be activated in the rea file for curved elements? If not, are there any general guidelines to using curved elements in this manner? Thanks, Jeremie -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Mon Oct 27 07:34:41 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 27 Oct 2014 13:34:41 +0100 Subject: [Nek5000-users] Restarting simulation in same run In-Reply-To: References: Message-ID: Hi Parveen There is implementation of arnoldi time-stepper for spectra calculation with parpack for nek5000. I think I've already sent you some info about it. The code you can find at ftp://ftp.mech.kth.se/pub/adam/Nek5000/toolbox/ There are sfd and arnoldi set-ups based on ext_cyl example form repository. The restart of the stepper you can find in the code. We do it by setting istep to 0 (forces solver to go through BDF1->BDF2->BDF3->BDF3) and resetting some vectors to zero. Best regards Adam On 22/10/14 15:07, nek5000-users at lists.mcs.anl.gov wrote: > Hello Paul > > I dont want to restart in the middle. > > Every time I want to start the time stepping from scratch. So I want > to know how to reset everything for a fresh simulation. Every time > ARPACK gives me a vector, I have to start with > BDF1->BDF2->BDF3->BDF3... until final time. > > I think accuracy of time stepper also matters since we are > approximating exact evolution operator exp(A*t) with the time stepper. > > I have read some papers on time stepping approach but the details are > not clear. > > Thanks > praveen > > On Wed, Oct 22, 2014 at 5:06 PM, > wrote: > > Hi Praveen, > > I would guess you need to develop your own timestepper because > Nek uses BDFk & EXTk to time march, where typically k=3. At > start-up, it uses k=1 and k=2 to bootstrap. If you start > timestepping > in the middle of the simulation then you need to overwrite the > existing coefficients so that you can restart the timestepper, > plus you need to store your initial condition (which presumably > amounts to overwriting vx,vy,vz, and pr). > > There are others in the community who have done these kinds of > things (I have not), who might have some further comments. > > Best, > > Paul > > PS ----- Are you trying to find eigenvalues associated with > the spatial operators ? i.e., does the accuracy of the > timestepper matter? If not, you can likely get away with > k=1, in which case you don't need much history in the backward > difference formula or extrapolation. Pressure, however, is > a bit of a tricky business because we use splitting and there > is a residual O(dt^k) impact... > > > > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Mon Oct 27 08:37:29 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 27 Oct 2014 19:07:29 +0530 Subject: [Nek5000-users] Restarting simulation in same run In-Reply-To: References: Message-ID: Thanks a lot. That will be a great help. Currently I am doing it calling nek via system function. Best regards Praveen On Monday, October 27, 2014, wrote: > Hi Parveen > > There is implementation of arnoldi time-stepper for spectra calculation > with parpack for nek5000. I think I've already sent you some info about it. > The code you can find at > ftp://ftp.mech.kth.se/pub/adam/Nek5000/toolbox/ > There are sfd and arnoldi set-ups based on ext_cyl example form > repository. The restart of the stepper you can find in the code. We do it > by setting istep to 0 (forces solver to go through BDF1->BDF2->BDF3->BDF3) > and resetting some vectors to zero. > Best regards > Adam > > On 22/10/14 15:07, nek5000-users at lists.mcs.anl.gov > wrote: > > Hello Paul > > I dont want to restart in the middle. > > Every time I want to start the time stepping from scratch. So I want to > know how to reset everything for a fresh simulation. Every time ARPACK > gives me a vector, I have to start with BDF1->BDF2->BDF3->BDF3... until > final time. > > I think accuracy of time stepper also matters since we are approximating > exact evolution operator exp(A*t) with the time stepper. > > I have read some papers on time stepping approach but the details are > not clear. > > Thanks > praveen > > On Wed, Oct 22, 2014 at 5:06 PM, > wrote: > >> Hi Praveen, >> >> I would guess you need to develop your own timestepper because >> Nek uses BDFk & EXTk to time march, where typically k=3. At >> start-up, it uses k=1 and k=2 to bootstrap. If you start timestepping >> in the middle of the simulation then you need to overwrite the >> existing coefficients so that you can restart the timestepper, >> plus you need to store your initial condition (which presumably >> amounts to overwriting vx,vy,vz, and pr). >> >> There are others in the community who have done these kinds of >> things (I have not), who might have some further comments. >> >> Best, >> >> Paul >> >> PS ----- Are you trying to find eigenvalues associated with >> the spatial operators ? i.e., does the accuracy of the >> timestepper matter? If not, you can likely get away with >> k=1, in which case you don't need much history in the backward >> difference formula or extrapolation. Pressure, however, is >> a bit of a tricky business because we use splitting and there >> is a residual O(dt^k) impact... >> > > > > > _______________________________________________ > Nek5000-users mailing listNek5000-users at lists.mcs.anl.gov https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Mon Oct 27 09:25:15 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 27 Oct 2014 10:25:15 -0400 Subject: [Nek5000-users] Curved GLL Points on Airfoil In-Reply-To: References: Message-ID: Hi Paul, I tried calling fix_geom in usrdat2 and it did not fix the problem. J?r?mie On Sunday, October 26, 2014, wrote: > > Hi Jeremie, > > Try calling "fix_geom" > > in usrdat2 after you modify the sides... this might provide the > requisite mesh clean up. > > Paul > > ------------------------------ > *From:* nek5000-users-bounces at lists.mcs.anl.gov > > [nek5000-users-bounces at lists.mcs.anl.gov > ] > on behalf of nek5000-users at lists.mcs.anl.gov > [ > nek5000-users at lists.mcs.anl.gov > ] > *Sent:* Sunday, October 26, 2014 4:10 PM > *To:* nek5000-users at lists.mcs.anl.gov > > *Subject:* [Nek5000-users] Curved GLL Points on Airfoil > > Hello Neks, > > > > I am trying to simulate the flow around an interpolated airfoil with > curved boundaries. To do so I am generating the collocation points > separately and writing them into an input file, which is read in the > usrdat2 subroutine before geometry re-initialization. The only curved > elements in the mesh are located at the airfoil boundary. I am not getting > any errors during geometry re-initialization. However, the calculation > starts off with a large courant number (around 7), which ends up diverging > after 5 times steps. Along the element edges near the airfoil, there seems > to be discontinuities in the flow which cause the solution to diverge. > However, when inputting non-curvilinear collocation points, written in the > exact same order to the input file, the simulation runs flawlessly. > > > > Are there any switches that must be activated in the rea file for curved > elements? > > > > If not, are there any general guidelines to using curved elements in this > manner? > > > > Thanks, > > > > Jeremie > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Mon Oct 27 10:47:10 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 27 Oct 2014 16:47:10 +0100 Subject: [Nek5000-users] Interfacing with Arpack In-Reply-To: References: , Message-ID: Hi Guiseppe I've just noticed you work on eigenvalue problem with arpack for nek5000 p_n-p_n. We've done similar work for p_n-p_n-2 and we are very interested in your work and we would like to share the code if it is possible. Our implementation of ext_cyl example you can find at ftp://ftp.mech.kth.se/pub/adam/Nek5000/toolbox/ Best regards Adam On 22/10/14 13:31, nek5000-users at lists.mcs.anl.gov wrote: > > Giuseppe, > > There are several masks: tmask() > > v1mask for vx > v2mask for vy > v3mask for vz > > pmask for pressure > > tmask has several fields in the situation where > you have different boundary conditions on different > passive scalars. > > Note that repeated application of one of the SEM operators > requires injection into H^1, which is effected through dssum > + mass matrix, as discussed earlier. Subroutine ALPHAM1 > in eigsolv illustrates this. (It's an older routine, so it > uses the multiplicity array when computing this inner products. > That would not be necessary, however, if the inner products > had been computed prior to dssum.) > > Paul > > > > > > On Wed, 22 Oct 2014, nek5000-users at lists.mcs.anl.gov wrote: > >> Hi Paul, thanks again for your reply. I think I am close to a >> solution now. One more thing: I think I need to mask out Dirichlet bc >> nodes when applying the matrix-vector products, right? How can I do >> this? Is it sufficient to call col2(u,vmask,n) after the >> matrix-vector product? >> Best, >> Giuseppe >> >> >> On 20/10/2014 16:37, nek5000-users at lists.mcs.anl.gov wrote: >>> Hi Giuseppe, >>> >>> You're close. >>> >>> The key point to understand is that application of Q^T arises from >>> continuity >>> requirements on the test functions (that we normally call "v" in an >>> abstract setting). >>> Everything in Nek is cast in the weak form: Find u \in X^N_0 such >>> that, for all v \in X^N_0, >>> >>> a( v, u )_N = a( v, u* )_N >>> >>> where u* is the exact solution and a( . , . )_N is the inner-product >>> of interest, with discrete >>> quadrature. For example, if >>> >>> a( v , u ) := \int_Omega grad v . grad u dV (1) >>> >>> then a( v , u )_N would be the same bilinear form with integration >>> replaced by numerical >>> quadrature. Note that an L2 projection would be of the form: >>> >>> >>> ( v , u ) = ( v , u* ) >>> >>> where I've dropped the _N on the inner product on the working >>> assumption that we >>> always use quadrature. >>> >>> Since u,v are continuous in Nek, the bilinear forms read: >>> >>> v^T Q^T M_L Q u = v^T Q^T M_L u*_L >>> >>> and, in nek, >>> >>> u = Binvm1 * QQ^T BM1 u* >>> >>> Note that (1), above, has the mass matrix build in because of the >>> volume >>> integral. So, technically, if you are doing dssum, you should >>> have a mass >>> matrix present, because dssum relates to projection against the test >>> functions >>> v, which implies a volume-weighted average of the equations. >>> >>> Sorry if this isn't very clear... >>> >>> Paul >>> >>> >>> >>> >>> ________________________________________ >>> From: nek5000-users-bounces at lists.mcs.anl.gov >>> [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of >>> nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >>> Sent: Monday, October 20, 2014 8:21 AM >>> To: nek5000-users at lists.mcs.anl.gov >>> Subject: Re: [Nek5000-users] Interfacing with Arpack >>> >>> Dear Paul, >>> thank you for the answer. >>> If my understanding is correct, since each matrix-vector product (coded >>> as matrix-matrix product) is done on the local variables, I should call >>> dssum after each matrix-vector operation. >>> For instance, if I want to apply the Helmholtz matrix to a vector u and >>> store the result in v, I should do: >>> call axhelm(v,u,h1,h2,imsh,1) >>> call dssum(v,nx1,ny1,nz1) >>> is it correct? >>> If I want to apply the gradient to a vector (always for eigenvalue >>> computation purposes) shoud I do something like: >>> call gradm1(dudx,dudy,dudz,u) >>> call dssum(dudx,nx1,ny1,nz1) >>> call dssum(dudy,nx1,ny1,nz1) >>> call dssum(dudz,nx1,ny1,nz1) >>> bufx = glsc2(u0,dudx,n) !bufx = u0.du/dx >>> or am I missing something? >>> >>> Giuseppe >>> >>> >>> >>> On 20/10/2014 12:58, nek5000-users at lists.mcs.anl.gov wrote: >>>> Giuseppe, >>>> >>>> The key thing to understand about Nek is that variables are >>>> _always_ stored in their local format, >>>> which implies you have redundant values at the interface such that >>>> they are ready for parallel operator >>>> evaluation. >>>> >>>> Let u_L denote such a vector with redundant values (L for Local) >>>> and let u be the corresponding >>>> vector without the redundantly stored data. Nominally, there is a >>>> matrix Q that maps u to u_L: >>>> >>>> (1) u_L = Q u >>>> >>>> and the matrix-vector product involving the mass matrix, of the >>>> form w = M u, can be expressed as: >>>> >>>> (2) w = M u = Q^T M_L Q u = Q^T M_L u_L >>>> >>>> where M_L = block_diag(M^e) , e=1,...,Nelv is the block-diagonal >>>> matrix consisting of local mass >>>> matrices within each element, e. In Nek, M^e is also block >>>> diagonal, so the whole _unassembled_ >>>> stiffness matrix, M_L, is diagonal. It is this matrix that is >>>> stored in bm1 ("B" on Mesh 1, the velocity/temp >>>> mesh). >>>> >>>> Notice that, as written, the output of (2) violates our "store only >>>> local vectors" policy, so we invoke >>>> (1) to correct this: >>>> >>>> >>>> (3) w_L = Q w = Q M u = Q Q^T M_L u_L. >>>> >>>> >>>> We refer to QQ^T as direct-stiffness summation (dssum, in nek). >>>> It is the operation that >>>> glues things together. >>>> >>>> So, suppose you wish to find a continuous function that is close to >>>> a discontinuous function >>>> in nek. Let f be the continuous thing and g be the discontinuous >>>> one. You can do this via: >>>> >>>> (4) f_L = Q M^-1 Q M_L g_L. >>>> >>>> In Nek, this would be written as: >>>> >>>> n = nx1*ny1*nz1*nelv >>>> >>>> call col3(f,g,bm1,n) >>>> >>>> call dssum(f,nx1,ny1,nz1) >>>> >>>> call col2 (f,binvm1,n) >>>> >>>> Here, binvm1 is effectively binvm1 = 1./ (QQT*bm1) , if we >>>> view bm1 and binvm1 as >>>> vectors, rather than diagonal matrices. >>>> >>>> Paul >>>> >>>> >>>> >>>> >>>> >>>> ________________________________________ >>>> From: nek5000-users-bounces at lists.mcs.anl.gov >>>> [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of >>>> nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >>>> Sent: Monday, October 20, 2014 2:03 AM >>>> To: nek5000-users at lists.mcs.anl.gov >>>> Subject: [Nek5000-users] Interfacing with Arpack >>>> >>>> Dear users and developers, >>>> I would like to solve a generalized eigenvalue problem with Arpack, >>>> based on Nek subroutines. Using reverse communication interface, I >>>> only >>>> need to express the action of the matrices on the vectors, but I am >>>> not >>>> sure on how to apply the mass matrix and the inverse mass matrix in >>>> the >>>> Pn-Pn formulation. >>>> Is it sufficient to write: >>>> call col2(p1,bm1,n) !apply mass matrix to vector p1 >>>> ... >>>> call invcol2(p1,bm1,n) ! apply inverse mass matrix to vector p1 >>>> or should I do: >>>> call rzero(h1,n) >>>> call rone(h2,n) >>>> call hmholtz('vel',p,p1,h1,h2,pmask,vmult,imsh,tolh,maxiter,isd) >>>> to apply inverse mass matrix to p1, or should I do something else? >>>> Also, from a related question from a couple of years ago, should I >>>> call >>>> col2(v,vmask,n) after each matrix-vector product? >>>> Thank you in advance. >>>> Best, >>>> Giuseppe >>>> _______________________________________________ >>>> Nek5000-users mailing list >>>> Nek5000-users at lists.mcs.anl.gov >>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>>> _______________________________________________ >>>> Nek5000-users mailing list >>>> Nek5000-users at lists.mcs.anl.gov >>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>> _______________________________________________ >>> Nek5000-users mailing list >>> Nek5000-users at lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>> _______________________________________________ >>> Nek5000-users mailing list >>> Nek5000-users at lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >> > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Mon Oct 27 11:17:47 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Mon, 27 Oct 2014 17:17:47 +0100 Subject: [Nek5000-users] Interfacing with Arpack In-Reply-To: References: , Message-ID: Hi Adam, thanks for sharing the code, I will study it, and if I can produce something useful I will contact you. Best regards, Giuseppe On 27/10/2014 16:47, nek5000-users at lists.mcs.anl.gov wrote: > Hi Guiseppe > > I've just noticed you work on eigenvalue problem with arpack for > nek5000 p_n-p_n. We've done similar work for p_n-p_n-2 and we are very > interested in your work and we would like to share the code if it is > possible. Our implementation of ext_cyl example you can find at > ftp://ftp.mech.kth.se/pub/adam/Nek5000/toolbox/ > Best regards > Adam > > On 22/10/14 13:31, nek5000-users at lists.mcs.anl.gov wrote: >> >> Giuseppe, >> >> There are several masks: tmask() >> >> v1mask for vx >> v2mask for vy >> v3mask for vz >> >> pmask for pressure >> >> tmask has several fields in the situation where >> you have different boundary conditions on different >> passive scalars. >> >> Note that repeated application of one of the SEM operators >> requires injection into H^1, which is effected through dssum >> + mass matrix, as discussed earlier. Subroutine ALPHAM1 >> in eigsolv illustrates this. (It's an older routine, so it >> uses the multiplicity array when computing this inner products. >> That would not be necessary, however, if the inner products >> had been computed prior to dssum.) >> >> Paul >> >> >> >> >> >> On Wed, 22 Oct 2014, nek5000-users at lists.mcs.anl.gov wrote: >> >>> Hi Paul, thanks again for your reply. I think I am close to a >>> solution now. One more thing: I think I need to mask out Dirichlet >>> bc nodes when applying the matrix-vector products, right? How can I >>> do this? Is it sufficient to call col2(u,vmask,n) after the >>> matrix-vector product? >>> Best, >>> Giuseppe >>> >>> >>> On 20/10/2014 16:37, nek5000-users at lists.mcs.anl.gov wrote: >>>> Hi Giuseppe, >>>> >>>> You're close. >>>> >>>> The key point to understand is that application of Q^T arises from >>>> continuity >>>> requirements on the test functions (that we normally call "v" in an >>>> abstract setting). >>>> Everything in Nek is cast in the weak form: Find u \in X^N_0 such >>>> that, for all v \in X^N_0, >>>> >>>> a( v, u )_N = a( v, u* )_N >>>> >>>> where u* is the exact solution and a( . , . )_N is the >>>> inner-product of interest, with discrete >>>> quadrature. For example, if >>>> >>>> a( v , u ) := \int_Omega grad v . grad u dV (1) >>>> >>>> then a( v , u )_N would be the same bilinear form with integration >>>> replaced by numerical >>>> quadrature. Note that an L2 projection would be of the form: >>>> >>>> >>>> ( v , u ) = ( v , u* ) >>>> >>>> where I've dropped the _N on the inner product on the working >>>> assumption that we >>>> always use quadrature. >>>> >>>> Since u,v are continuous in Nek, the bilinear forms read: >>>> >>>> v^T Q^T M_L Q u = v^T Q^T M_L u*_L >>>> >>>> and, in nek, >>>> >>>> u = Binvm1 * QQ^T BM1 u* >>>> >>>> Note that (1), above, has the mass matrix build in because of the >>>> volume >>>> integral. So, technically, if you are doing dssum, you should >>>> have a mass >>>> matrix present, because dssum relates to projection against the >>>> test functions >>>> v, which implies a volume-weighted average of the equations. >>>> >>>> Sorry if this isn't very clear... >>>> >>>> Paul >>>> >>>> >>>> >>>> >>>> ________________________________________ >>>> From: nek5000-users-bounces at lists.mcs.anl.gov >>>> [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of >>>> nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >>>> Sent: Monday, October 20, 2014 8:21 AM >>>> To: nek5000-users at lists.mcs.anl.gov >>>> Subject: Re: [Nek5000-users] Interfacing with Arpack >>>> >>>> Dear Paul, >>>> thank you for the answer. >>>> If my understanding is correct, since each matrix-vector product >>>> (coded >>>> as matrix-matrix product) is done on the local variables, I should >>>> call >>>> dssum after each matrix-vector operation. >>>> For instance, if I want to apply the Helmholtz matrix to a vector u >>>> and >>>> store the result in v, I should do: >>>> call axhelm(v,u,h1,h2,imsh,1) >>>> call dssum(v,nx1,ny1,nz1) >>>> is it correct? >>>> If I want to apply the gradient to a vector (always for eigenvalue >>>> computation purposes) shoud I do something like: >>>> call gradm1(dudx,dudy,dudz,u) >>>> call dssum(dudx,nx1,ny1,nz1) >>>> call dssum(dudy,nx1,ny1,nz1) >>>> call dssum(dudz,nx1,ny1,nz1) >>>> bufx = glsc2(u0,dudx,n) !bufx = u0.du/dx >>>> or am I missing something? >>>> >>>> Giuseppe >>>> >>>> >>>> >>>> On 20/10/2014 12:58, nek5000-users at lists.mcs.anl.gov wrote: >>>>> Giuseppe, >>>>> >>>>> The key thing to understand about Nek is that variables are >>>>> _always_ stored in their local format, >>>>> which implies you have redundant values at the interface such that >>>>> they are ready for parallel operator >>>>> evaluation. >>>>> >>>>> Let u_L denote such a vector with redundant values (L for Local) >>>>> and let u be the corresponding >>>>> vector without the redundantly stored data. Nominally, there is a >>>>> matrix Q that maps u to u_L: >>>>> >>>>> (1) u_L = Q u >>>>> >>>>> and the matrix-vector product involving the mass matrix, of the >>>>> form w = M u, can be expressed as: >>>>> >>>>> (2) w = M u = Q^T M_L Q u = Q^T M_L u_L >>>>> >>>>> where M_L = block_diag(M^e) , e=1,...,Nelv is the block-diagonal >>>>> matrix consisting of local mass >>>>> matrices within each element, e. In Nek, M^e is also block >>>>> diagonal, so the whole _unassembled_ >>>>> stiffness matrix, M_L, is diagonal. It is this matrix that is >>>>> stored in bm1 ("B" on Mesh 1, the velocity/temp >>>>> mesh). >>>>> >>>>> Notice that, as written, the output of (2) violates our "store >>>>> only local vectors" policy, so we invoke >>>>> (1) to correct this: >>>>> >>>>> >>>>> (3) w_L = Q w = Q M u = Q Q^T M_L u_L. >>>>> >>>>> >>>>> We refer to QQ^T as direct-stiffness summation (dssum, in nek). >>>>> It is the operation that >>>>> glues things together. >>>>> >>>>> So, suppose you wish to find a continuous function that is close >>>>> to a discontinuous function >>>>> in nek. Let f be the continuous thing and g be the discontinuous >>>>> one. You can do this via: >>>>> >>>>> (4) f_L = Q M^-1 Q M_L g_L. >>>>> >>>>> In Nek, this would be written as: >>>>> >>>>> n = nx1*ny1*nz1*nelv >>>>> >>>>> call col3(f,g,bm1,n) >>>>> >>>>> call dssum(f,nx1,ny1,nz1) >>>>> >>>>> call col2 (f,binvm1,n) >>>>> >>>>> Here, binvm1 is effectively binvm1 = 1./ (QQT*bm1) , if we >>>>> view bm1 and binvm1 as >>>>> vectors, rather than diagonal matrices. >>>>> >>>>> Paul >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> ________________________________________ >>>>> From: nek5000-users-bounces at lists.mcs.anl.gov >>>>> [nek5000-users-bounces at lists.mcs.anl.gov] on behalf of >>>>> nek5000-users at lists.mcs.anl.gov [nek5000-users at lists.mcs.anl.gov] >>>>> Sent: Monday, October 20, 2014 2:03 AM >>>>> To: nek5000-users at lists.mcs.anl.gov >>>>> Subject: [Nek5000-users] Interfacing with Arpack >>>>> >>>>> Dear users and developers, >>>>> I would like to solve a generalized eigenvalue problem with Arpack, >>>>> based on Nek subroutines. Using reverse communication interface, I >>>>> only >>>>> need to express the action of the matrices on the vectors, but I >>>>> am not >>>>> sure on how to apply the mass matrix and the inverse mass matrix >>>>> in the >>>>> Pn-Pn formulation. >>>>> Is it sufficient to write: >>>>> call col2(p1,bm1,n) !apply mass matrix to vector p1 >>>>> ... >>>>> call invcol2(p1,bm1,n) ! apply inverse mass matrix to vector p1 >>>>> or should I do: >>>>> call rzero(h1,n) >>>>> call rone(h2,n) >>>>> call hmholtz('vel',p,p1,h1,h2,pmask,vmult,imsh,tolh,maxiter,isd) >>>>> to apply inverse mass matrix to p1, or should I do something else? >>>>> Also, from a related question from a couple of years ago, should I >>>>> call >>>>> col2(v,vmask,n) after each matrix-vector product? >>>>> Thank you in advance. >>>>> Best, >>>>> Giuseppe >>>>> _______________________________________________ >>>>> Nek5000-users mailing list >>>>> Nek5000-users at lists.mcs.anl.gov >>>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>>>> _______________________________________________ >>>>> Nek5000-users mailing list >>>>> Nek5000-users at lists.mcs.anl.gov >>>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>>> _______________________________________________ >>>> Nek5000-users mailing list >>>> Nek5000-users at lists.mcs.anl.gov >>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>>> _______________________________________________ >>>> Nek5000-users mailing list >>>> Nek5000-users at lists.mcs.anl.gov >>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>> >>> _______________________________________________ >>> Nek5000-users mailing list >>> Nek5000-users at lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users >>> >> _______________________________________________ >> Nek5000-users mailing list >> Nek5000-users at lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users From nek5000-users at lists.mcs.anl.gov Tue Oct 28 07:12:51 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 28 Oct 2014 12:12:51 +0000 Subject: [Nek5000-users] Submitting Job Nek Enabled Moab Message-ID: Hi All, I am currently experiencing issues when trying to run jobs on my hpc system. I have installed moab (with hdf5 and zoltan) as recommended on the nek5000 webpages - https://nek5000.mcs.anl.gov/index.php/Building_and_Using_Nek_/_MOAB. When I do this the simulation will run if I acquire resources form the system and then run the job using nekmpi [casename] [np]. However if I then attempt to submit this same simulation to the queue using: #!/bin/bash #PBS -l nodes=1:ppn=12,pvmem=1900mb,walltime=01:00:00 #PBS -V cd $PBS_O_WORKDIR nekmpi pipe 8 > log exit 0 I submit the job and it fails citing: mv: cannot stat `pipe.log.8': No such file or directory mv: cannot stat `pipe.sch': No such file or directory node6.8452Error re-mmapping shared memory: Cannot allocate memory (err=9) node6.8452Error re-mmapping shared memory: Cannot allocate memory [node6:08452] Open MPI detected an unexpected PSM error in opening an endpoint: Error re-mmapping shared memory: Cannot allocate memory -------------------------------------------------------------------------- mpiexec has exited due to process rank 7 with PID 8452 on node node6 exiting without calling "finalize". This may have caused other processes in the application to be terminated by signals sent by mpiexec (as reported here). -------------------------------------------------------------------------- I thus decided there may be issues with moab so I installed according to the moab webpages - http://trac.mcs.anl.gov/projects/ITAPS/wiki/BuildingAndUsingMOAB. When I submit the job now I see a different error: mv: cannot stat `pipe.log.8': No such file or directory mv: cannot stat `pipe.sch': No such file or directory ./nek5000: error while loading shared libraries: libsz.so.2: cannot open shared object file: No such file or directory The above error I feel is somehow linked to either netcdf or hdf5 but I cannot see how or why as they both installed correctly (as far as I can tell). Thank you for any help, Friedrich -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Tue Oct 28 08:29:00 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Tue, 28 Oct 2014 08:29:00 -0500 Subject: [Nek5000-users] Submitting Job Nek Enabled Moab In-Reply-To: References: Message-ID: Friedrich, Make sure your LD_LIBRARY_PATH has the path to szip libraries. If you do "ldd nek5000", you should be able to see whether the shared libraries are resolved at runtime. Make sure that all libraries are loaded before running nek5000 driver. Vijay On Tue, Oct 28, 2014 at 7:12 AM, wrote: > Hi All, > > > I am currently experiencing issues when trying to run jobs on my hpc system. > > > I have installed moab (with hdf5 and zoltan) as recommended on the nek5000 > webpages - > https://nek5000.mcs.anl.gov/index.php/Building_and_Using_Nek_/_MOAB. > > > When I do this the simulation will run if I acquire resources form the > system and then run the job using nekmpi [casename] [np]. However if I then > attempt to submit this same simulation to the queue using: > > > #!/bin/bash > > #PBS -l nodes=1:ppn=12,pvmem=1900mb,walltime=01:00:00 > > #PBS -V > > > cd $PBS_O_WORKDIR > > nekmpi pipe 8 > log > > exit 0 > > > I submit the job and it fails citing: > > > mv: cannot stat `pipe.log.8': No such file or directory > mv: cannot stat `pipe.sch': No such file or directory > node6.8452Error re-mmapping shared memory: Cannot allocate memory (err=9) > node6.8452Error re-mmapping shared memory: Cannot allocate memory > [node6:08452] Open MPI detected an unexpected PSM error in opening an > endpoint: Error re-mmapping shared memory: Cannot allocate memory > -------------------------------------------------------------------------- > mpiexec has exited due to process rank 7 with PID 8452 on > node node6 exiting without calling "finalize". This may > have caused other processes in the application to be > terminated by signals sent by mpiexec (as reported here). > -------------------------------------------------------------------------- > > > I thus decided there may be issues with moab so I installed according to the > moab webpages - > http://trac.mcs.anl.gov/projects/ITAPS/wiki/BuildingAndUsingMOAB. > > > When I submit the job now I see a different error: > > > mv: cannot stat `pipe.log.8': No such file or directory > mv: cannot stat `pipe.sch': No such file or directory > ./nek5000: error while loading shared libraries: libsz.so.2: cannot open > shared object file: No such file or directory > > > The above error I feel is somehow linked to either netcdf or hdf5 but I > cannot see how or why as they both installed correctly (as far as I can > tell). > > > Thank you for any help, > > > Friedrich > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > From nek5000-users at lists.mcs.anl.gov Wed Oct 29 07:52:48 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 29 Oct 2014 12:52:48 +0000 Subject: [Nek5000-users] Submitting Job Nek Enabled Moab Message-ID: Hi Vijay, Ok I have done this, thank you for the advice! As I am submitting from a batch script should I consider running using nekpbs? If so what edits to the script would this entail etc. Friedrich -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Wed Oct 29 10:37:41 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 29 Oct 2014 15:37:41 +0000 Subject: [Nek5000-users] Submitting Job Nek Enabled Moab Message-ID: Hi Again, I just wanted to update that although I have added the szip library to the LD_LIBRARY_PATH I still receive the same error: mv: cannot stat `pipe.log.12': No such file or directory mv: cannot stat `pipe.sch': No such file or directory ./nek5000: error while loading shared libraries: libintlc.so.5: failed to map segment from shared object: Cannot allocate memory ./nek5000: error while loading shared libraries: libintlc.so.5: failed to map segment from shared object: Cannot allocate memory ./nek5000: error while loading shared libraries: libintlc.so.5: failed to map segment from shared object: Cannot allocate memory ./nek5000: error while loading shared libraries: libintlc.so.5: failed to map segment from shared object: Cannot allocate memory Should I consider loading zlib etc in the same way or is this something else? Regards, Friedrich. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nek5000-users at lists.mcs.anl.gov Wed Oct 29 11:08:01 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Wed, 29 Oct 2014 11:08:01 -0500 Subject: [Nek5000-users] Submitting Job Nek Enabled Moab In-Reply-To: References: Message-ID: Friedrich, If you are not sure about how to compile/link/run with shared libraries, it is in general advisable to use static libraries everywhere with position-independent-code (--with-pic or CFLAGS, FCFLAGS="-fPIC"). This would remove any headaches about symbols not getting resolved during runtime. Alternately, if you do want to use shared libraries, then make sure LD_LIBRARY_PATH has all the relevant paths to the libraries so that the runtime loader can resolve the symbols correctly. To check, try ldd ./nek5000 | grep "not found" If all the libraries are resolved (no output from above command), then please send us the architecture, compiler info (intel ?), HDF5 config.log, MOAB config.log so that we can find out the cause of the issues. > ./nek5000: error while loading shared libraries: libintlc.so.5: failed to > map segment from shared object: Cannot allocate memory This is indicative of hitting shared memory limit -- and I can't really place whether this is due to your job submission system or system configuration. Can you try running interactively and see if you are still getting the errors below ? Vijay On Wed, Oct 29, 2014 at 10:37 AM, wrote: > Hi Again, > > > I just wanted to update that although I have added the szip library to the > LD_LIBRARY_PATH I still receive the same error: > > > mv: cannot stat `pipe.log.12': No such file or directory > mv: cannot stat `pipe.sch': No such file or directory > ./nek5000: error while loading shared libraries: libintlc.so.5: failed to > map segment from shared object: Cannot allocate memory > ./nek5000: error while loading shared libraries: libintlc.so.5: failed to > map segment from shared object: Cannot allocate memory > ./nek5000: error while loading shared libraries: libintlc.so.5: failed to > map segment from shared object: Cannot allocate memory > ./nek5000: error while loading shared libraries: libintlc.so.5: failed to > map segment from shared object: Cannot allocate memory > > > Should I consider loading zlib etc in the same way or is this something > else? > > > Regards, Friedrich. > > > _______________________________________________ > Nek5000-users mailing list > Nek5000-users at lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users > From nek5000-users at lists.mcs.anl.gov Thu Oct 30 07:44:57 2014 From: nek5000-users at lists.mcs.anl.gov (nek5000-users at lists.mcs.anl.gov) Date: Thu, 30 Oct 2014 12:44:57 +0000 Subject: [Nek5000-users] Submitting Job Nek Enabled Moab Message-ID: Hi Vijay, Firstly thank you for your response! I have attempted all you suggested and this still does not seem to fix my issues. When I run ldd nek5000 I see: linux-vdso.so.1 => (0x00007fff635ff000) libz.so.1 => /home/eng/esumgy/pre-moab/zlib/zlib-1.2.8/libz.so.1 (0x00007fdd5a627000) libm.so.6 => /lib64/libm.so.6 (0x00007fdd5a37a000) libsz.so.2 => /home/eng/esumgy/pre-moab/szip/szip-2.1/gcc/lib/libsz.so.2 (0x00007fdd5a166000) libgpfs.so => /usr/lib64/libgpfs.so (0x00007fdd5a05a000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fdd59e55000) libmpi_f77.so.0 => /hpcwarwick/ompi/1.4.5/intel/11.1/lib/libmpi_f77.so.0 (0x00007fdd59c19000) libmpi.so.0 => /hpcwarwick/ompi/1.4.5/intel/11.1/lib/libmpi.so.0 (0x00007fdd59945000) libopen-rte.so.0 => /hpcwarwick/ompi/1.4.5/intel/11.1/lib/libopen-rte.so.0 (0x00007fdd596dc000) libopen-pal.so.0 => /hpcwarwick/ompi/1.4.5/intel/11.1/lib/libopen-pal.so.0 (0x00007fdd5945e000) libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fdd59246000) libutil.so.1 => /lib64/libutil.so.1 (0x00007fdd59042000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fdd58d3c000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdd58b26000) libintlc.so.5 => /gpfs/hpcwarwick/intel/Compiler/11.1/059/lib/intel64/libintlc.so.5 (0x00007fdd589e8000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdd587cb000) libc.so.6 => /lib64/libc.so.6 (0x00007fdd58452000) librt.so.1 => /lib64/librt.so.1 (0x00007fdd58248000) /lib64/ld-linux-x86-64.so.2 (0x00007fdd5a83f000) libimf.so => /gpfs/hpcwarwick/intel/Compiler/11.1/059/lib/intel64/libimf.so (0x00007fdd57eb5000) libsvml.so => /gpfs/hpcwarwick/intel/Compiler/11.1/059/lib/intel64/libsvml.so (0x00007fdd57c9d000) The compiler being used is ompi/1.4.5/intel/11.1 See also the moab and hdf5 config.log attached! System information can be found - http://www2.warwick.ac.uk/fac/sci/csc/local/hpc Thank you for your help I am happy maybe to finally solve this issue Friedrich -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hdf5_config.log.zip Type: application/zip Size: 29221 bytes Desc: hdf5_config.log.zip URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: moab_config.log.zip Type: application/zip Size: 22251 bytes Desc: moab_config.log.zip URL: