[Nek5000-users] Proper setup for AMG solver

Mon Sep 10 07:54:04 CDT 2018

Hi Steffen,

Sorry for the delayed response.

My feeling re. characteristics + 10 PS is the following.

A)  characteristics is designed to trade off more work on advection for less work on the implicit part.  The principal implicit cost in NS is the pressure solve.    Typically, we see that we spend 50% of our time in advection when using the characteristics scheme (but this can vary, depending on whether you go for 2nd-order (recommended, as Stefan suggested) or 3rd-order in time (not recommend for characteristics), and on the amount of over-integration for the advection - typically lxd=3*lx1/2, but you can get away with less and might want to do so for characteristics.

In the case of multiple PS, you increase your advection work proportionally, but the pressure work stays the same.  Thus, an approach that puts more emphasis on the expensive part will ultimately not be a winning strategy.

Just to give you some estimates:

Standard advection (IFCHAR=F):

    1 advection evaluation per time step for each component of velocity and for each passive scalar:

Advection work unit:  W ~ 18(MN)^2 ops, where M=3/2N for

standard dealiasing. (This is a crude estimate.)

So, IFCHAR=F --> (3+PS)*W advection work per step, for "PS"

passive scalars, including temperature.

With IFCHAR=T, assume you are running at a CFL of 2.0 (with

p26==1, which implies one RK4 step to achieve CFL=2).

For 2nd-order, there are two RK4 steps, each requiring 4 sub steps,

so the advection work is 8 x (3+PS)*W.

If you run 3rd-order, it is 12x(3+PS)*W because 3rd-order characteristics requires 3 RK4 steps.   More details can be found in some old notes posted here:

http://www.mcs.anl.gov/~fischer/oifs.pdf

and in a forthcoming paper that's currently under review.

B)  If the time savings is not significant, I generally prefer to set IFCHAR=F and then run 3rd order.    The std. BDFk/EXTk (IFCHAR=F) case requires only (3+PS)W advection work per step, independent of temporal order.

hth

Paul

________________________________
From: Nek5000-users <nek5000-users-bounces at lists.mcs.anl.gov> on behalf of nek5000-users at lists.mcs.anl.gov <nek5000-users at lists.mcs.anl.gov>
Sent: Monday, September 10, 2018 6:12:49 AM
To: nek5000-users at lists.mcs.anl.gov
Subject: Re: [Nek5000-users] Proper setup for AMG solver

The slow AMG problem he reported was related to an old (buggy) version of amg_hypre. Note, in my experience AMG is faster at least for non-power-of-two ranks (quite common today) independent of the problem size.

Regarding OIFS it’s important to use BDF2 with a possible low lxd otherwise it’s quite expensive. It depends heavily on how expensive the pressure solve is.

Cheers
Stefan

> On 10 Sep 2018, at 12:51, "nek5000-users at lists.mcs.anl.gov" <nek5000-users at lists.mcs.anl.gov> wrote:
>
> Hi Steffen,
> It is interesting that you find that the characteristic scheme is indeed
> faster; similar tests with a a pipe (a few years ago) have shown the
> opposite, which is why we ran our 2013 pipe using conventional time
> steppers. But I guess we should re-do these tests!
>
> I think your question regarding the temporal evolution is interesting,
> and I guess in the end this has to be judged using physical time scales
> (which scale in plus units for the velocity here). Not being too
> familiar with the characteristic scheme, what is really the smallest
> resolved time scale? Is it the actual time step, or the RK steps, or
> some intermediate scale?
>
> We have found that AMG is indeed surpassing XXT at some point, but I
> guess this is machine dependent (and number of processors in addition to
> number of elements). Do you have any issues with the setup of XXT taking
> quite some time?
>
> Best,
> Philipp
>
>
>> On 2018-09-02 12:05, nek5000-users at lists.mcs.anl.gov wrote:
>> Hello Stefan & Paul,
>>
>> thanks for your suggestions. I use Nek5000 v17 (mver 17.0.4 in makenek).
>>
>> I have taken a closer look at the logfiles as suggested by Paul. It seems like I spend the most time for the scalar fields second most for the velocity fields and pressure solve is the smallest share. The numbers below are in seconds for a typical timestep without calculating statistics or writing out files at Re_b = 5300 (Re_t = u_t D / nu = 360, 768 cores) and Re_b = 37700 (Re_t = 2000, 6144 cores).
>> I use projection for all 10 scalar fields.
>>
>> 360:
>> Scalars done        0.064
>> Fluid done        0.039
>> U-PRES gmres        0.021
>> Step            0.127
>>
>> 2000:
>> Scalars done        0.94
>> Fluid done        0.51
>> U-PRES gmres        0.21
>> Step            1.72
>>
>>
>> Paul, could you elaborate on why you would not use characteristics when running 10 scalar fields?
>> In my short tests at Re_b=5300, it appears to be more time consuming per step but due to the increase in DT it is worth it to go to characteristics. The time per timestep for targetCFL=2.0 increases by a factor of 3 but DT is increased by a factor of 6.
>> Besides, are such increased timesteps still small enough to capture the temporal evolution of the flow?
>>
>>
>> When I am back on my workstation, I will create a different mesh with lx1=8, lxd=10 and run with the settings Stefan suggested. I expect to get a significant speedup when using the lower tolerances also for velocity and scalars and changing to characteristics.
>>
>> I know about the limit of 350k elements for XXT as I have once commented out the part of the code where this is tested.
>> Since for my setups AMG was always slower than XXT, I am thinking about sticking to XXT.
>> Is there any other reason than your experience with AMG and XXT at large number of elements to enforce AMG?
>>
>>
>> Best Regards,
>> Steffen
>>
>>
>>
>> Message: 5
>> Date: Sat, 1 Sep 2018 17:10:54 +0200
>> From: nek5000-users at lists.mcs.anl.gov
>> To: nek5000-users at lists.mcs.anl.gov     <nek5000-users at lists.mcs.anl.gov>
>> Subject: Re: [Nek5000-users] Proper setup for AMG solver
>> Message-ID:
>>         <mailman.9805.1535814716.86639.nek5000-users at lists.mcs.anl.gov>
>> Content-Type: text/plain; charset=utf-8
>>
>> Try to use
>>
>> lx1=8/lxd=10 with a (potentially) finer mesh
>> BDF2 + OIFS with a targetCFL=3.5
>> set dt = 0 (this will adjust dt to targetCFL)
>> pressure tol = 1e-5 (residual projection turned on) 1e-6 for velocity and scalars (residual projection turned off)
>>
>> Note, a mesh with more than 350k elements requires AMG. The default parameters are fine.
>>
>> What version of Nek5000 are you using?
>>
>> Cheers,
>> Stefan
>>
>>
>> -----Original message-----
>>> From:nek5000-users at lists.mcs.anl.gov <nek5000-users at lists.mcs.anl.gov>
>>> Sent: Saturday 1st September 2018 16:50
>>> To: nek5000-users at lists.mcs.anl.gov
>>> Subject: [Nek5000-users] Proper setup for AMG solver
>>>
>>> Dear Nek users & experts,
>>>
>>> I am currently running a turbulent pipe flow very similar to the simulations from El Khoury et al 2013.
>>>
>>> Additionally, I solve for 10 thermal fields being treated as passive scalars.
>>>
>>> The Reynolds number is the same as the highest in El Khoury (2013), Re_b = 37700.
>>>
>>> As I am using the relaxation term filtering (RT-Filter), I have a slightly lower resolution of about 250,000 elements at N=11 (5 times less than El Khoury).
>>>
>>> As the simulation is still very heavy, I have been looking into ways for speeding it up.
>>> I found some good suggestions here:
>>> http://nek5000.github.io/NekDoc/faq.html?highlight=amg#computational-speed
>>>
>>> and here (older version?)
>>> http://nek5000.github.io/NekDoc/large_scale.html
>>>
>>>
>>> However, I have some questions regarding these suggestions.
>>> 1) Dealiasing:
>>> Usually I use lxd = 3/2*lx1. Can I lower that or even use lxd=lx1?
>>>
>>> 2) Tolerances:
>>> I have tested to reduce the tolerance for pressure from 1e-8 to 5e-5 for a run at Re_b=5300 without any significant speedup. Would you consider 5e-5 for pressure accurate enough for evaluating statistics like turbulent kinetic energy budgets, Reynolds shear
>>>  stress budgets or budget of turbulent heat fluxes?
>>>
>>> 3) Time discretisation: BDF2 and OIFS with Courant=2-5
>>> If I go from BDF3/EXT3 at C=0.5 to BDF2/OIFS at C=5.0, will I not miss high frequency fluctuations in time, since DT is much larger?
>>>
>>> 4) AMG instead of XXT:
>>> I have tested AMG instead of XXT for both Re_b=5300 and Re_b=37700 without any speedup. Time/timestep is even higher with AMG. My workflow looks like this
>>> 4.1) Set SEMG_AMG in the par file.
>>> 4.2) Run the simulation once to dump amg files.
>>> 4.3) Run amg_hypre (Here I do not know which options to choose, thus I have only uses default settings)
>>> 4.4) Run the simulation.
>>> Maybe I should choose different options for amg_hypre, or should I rather use the amg_matlab2 tools? For the matlab tools I have not found an explanation on how to use them.
>>>
>>>
>>> I am grateful for any advice regarding these aspects.
>>>
>>> Best Regards,
>>> Steffen
>>>
>>>
>>>
>>> _______________________________________________
>>> Nek5000-users mailing list
>>> Nek5000-users at lists.mcs.anl.gov
>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>>
>>
>> ------------------------------
>>
>> Message: 6
>> Date: Sat, 1 Sep 2018 15:15:29 +0000
>> From: nek5000-users at lists.mcs.anl.gov
>> To: "nek5000-users at lists.mcs.anl.gov"
>>         <nek5000-users at lists.mcs.anl.gov>
>> Subject: Re: [Nek5000-users] Proper setup for AMG solver
>> Message-ID:
>>         <mailman.9806.1535814932.86639.nek5000-users at lists.mcs.anl.gov>
>> Content-Type: text/plain; charset="us-ascii"
>>
>>
>> How much time are you spending in your scalar fields?
>>
>>
>> Do you have projection turned on for all these fields?
>>
>>
>> grep tep logfile
>>
>>
>> will tell you how much time per step
>>
>>
>> grep gmr logfile will tell you how much time in the pressure on each step
>>
>>
>> What's left over is mostly passive scalar, unless you are using characteristics.
>>
>>
>> I would not recommend characteristics when running 10 scalar fields.
>>
>>
>> Paul
>>
>>
>> ________________________________
>> From: Nek5000-users <nek5000-users-bounces at lists.mcs.anl.gov> on behalf of nek5000-users at lists.mcs.anl.gov <nek5000-users at lists.mcs.anl.gov>
>> Sent: Saturday, September 1, 2018 9:50:38 AM
>> To: nek5000-users at lists.mcs.anl.gov
>> Subject: [Nek5000-users] Proper setup for AMG solver
>>
>>
>> Dear Nek users & experts,
>>
>> I am currently running a turbulent pipe flow very similar to the simulations from El Khoury et al 2013.
>> Additionally, I solve for 10 thermal fields being treated as passive scalars.
>>
>> The Reynolds number is the same as the highest in El Khoury (2013), Re_b = 37700.
>> As I am using the relaxation term filtering (RT-Filter), I have a slightly lower resolution of about 250,000 elements at N=11 (5 times less than El Khoury).
>>
>> As the simulation is still very heavy, I have been looking into ways for speeding it up.
>> I found some good suggestions here:
>> http://nek5000.github.io/NekDoc/faq.html?highlight=amg#computational-speed
>>
>> and here (older version?)
>> http://nek5000.github.io/NekDoc/large_scale.html
>>
>>
>> However, I have some questions regarding these suggestions.
>> 1) Dealiasing:
>> Usually I use lxd = 3/2*lx1. Can I lower that or even use lxd=lx1?
>>
>> 2) Tolerances:
>> I have tested to reduce the tolerance for pressure from 1e-8 to 5e-5 for a run at Re_b=5300 without any significant speedup. Would you consider 5e-5 for pressure accurate enough for evaluating statistics like turbulent kinetic energy budgets, Reynolds shear stress budgets or budget of turbulent heat fluxes?
>>
>> 3) Time discretisation: BDF2 and OIFS with Courant=2-5
>> If I go from BDF3/EXT3 at C=0.5 to BDF2/OIFS at C=5.0, will I not miss high frequency fluctuations in time, since DT is much larger?
>>
>> 4) AMG instead of XXT:
>> I have tested AMG instead of XXT for both Re_b=5300 and Re_b=37700 without any speedup. Time/timestep is even higher with AMG. My workflow looks like this
>> 4.1) Set SEMG_AMG in the par file.
>> 4.2) Run the simulation once to dump amg files.
>> 4.3) Run amg_hypre (Here I do not know which options to choose, thus I have only uses default settings)
>> 4.4) Run the simulation.
>> Maybe I should choose different options for amg_hypre, or should I rather use the amg_matlab2 tools? For the matlab tools I have not found an explanation on how to use them.
>>
>>
>> I am grateful for any advice regarding these aspects.
>>
>> Best Regards,
>> Steffen
>>
>>
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://lists.mcs.anl.gov/pipermail/nek5000-users/attachments/20180901/62611526/attachment.html>
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> Nek5000-users mailing list
>> Nek5000-users at lists.mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>>
>>
>> ------------------------------
>>
>> End of Nek5000-users Digest, Vol 115, Issue 1
>> *********************************************
>> _______________________________________________
>> Nek5000-users mailing list
>> Nek5000-users at lists.mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>>
> _______________________________________________
> Nek5000-users mailing list
> Nek5000-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users

_______________________________________________
Nek5000-users mailing list
Nek5000-users at lists.mcs.anl.gov
https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/nek5000-users/attachments/20180910/79b74496/attachment-0001.html>