[Nek5000-users] Significant computational time increase from 2D to 3D runs

[nek5000-users at lists.mcs.anl.gov]

Dear Kat,


When you say 9-points per element, are you running with lx1=3?


Let me state that this is really suboptimal for Nek, which was designed for lx1 = 8-20, with lx1=5 about as low as I would go for any reasonable calculation.


That being said, your analysis is somewhat correct, but there are a few other things to be concerned with in going from 2D to 3D.


You start with 2D on a 4x100 mesh, 9 points per element.


Going to 3D, you have 4x4x100, 3x9 points per element, so an increase of 12x.


In addition, many of the operations go from 2x2 to 3x3, which is a ratio of 2.25.  (e.g., the Laplacian now involves more work as you have derivatives with respect to r,s, and t in the local coordinates and their transposes, etc.).  In addition to each operation being more expensive per gridpiont, you now have 3 velocity components instead of 2, so there is another multiplier there.   Moreover, there are more surface points, 6 x 3^2, instead of 4 x 3, so 54 instead of 12, for each element.


My suggestion would be to first rethink your 2D and 3D problems in terms of 8x8 = 64 or 8x8x8=512 points per element,  at the target resolution.   Then, start the calculation with lx1=5 or 6 to converge to a reasonable solution and restart with at the target value.


hth,


Paul


________________________________
From: Nek5000-users <nek5000-users-bounces at lists.mcs.anl.gov> on behalf of nek5000-users at lists.mcs.anl.gov <nek5000-users at lists.mcs.anl.gov>
Sent: Wednesday, March 21, 2018 11:45:35 AM
To: nek5000-users at lists.mcs.anl.gov
Subject: [Nek5000-users] Significant computational time increase from 2D to 3D runs

Hello,

I am running a DNS cross-flow, two density fluid, turbulent duct simulation and I see a significant increase in computation time going from 2D to 3D that cannot be attributed to a simple increase in grid points. I believe I am at the small end of simulation size for Nek5000, as I am only running on average 4-500k grid points and running on 1-32 cores. Keeping all other things the same (i.e., parameters, physical set-up, etc), I’ve run a 2D simulation of 100x4 elements and a 3D simulation with 100x4x4 elements, both with 9 grid points per element and on 4 cores, and found a increase in computational time of more than 100 fold. I’ve also run 2D simulations of 200x16 elements on 4 cores (again everything else the same) and found it still ran ~100 times faster than the 3D simulations that had half the amount of grid points. This doesn’t make sense to me and implies that this issue would not be simply resolved by following the “Performance and Memory Considerations” advice in the Nek5000 Docs as it is not just about an increase in grid points. It seems to be linked to going from 2D to 3D, rather than due to an increase in grid points.

I feel that I am missing something fundamental here. Is there something one should do when switching from 2D to 3D in Nek5000 that I am not aware of? I have searched through old email threads but could not find anything that mentions something like this.

Thanks!
- Kat
_______________________________________________
Nek5000-users mailing list
Nek5000-users at lists.mcs.anl.gov
https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/nek5000-users/attachments/20180321/bc37e734/attachment.html>