[Nek5000-users] run-time hang up in gs_setup
nek5000-users at lists.mcs.anl.gov
nek5000-users at lists.mcs.anl.gov
Thu Jan 12 13:38:17 CST 2012
Hi Stefan;
Unfortunately it does not work. first of all, ./run in amg_matlab does
not produce anything (latest version). Using my version of the files
(running turbChannel), the simulation crashed giving me this error:
###############################################
AMG level 8: 3 iterations with rho = 0.680429
AMG level 9: 2 iterations with rho = 0.560188
AMG: 144540 rows
AMG: reading through row 1200, pass 1/121
ERROR (proc 0000,
/scratch/azad/codes/late/nek5_svn/trunk/nek/jl/amg.c:875): AMG: data
has more rows than given problem
call exitt: dying ...
###############################################
Regards
Azad
Quoting nek5000-users-request at lists.mcs.anl.gov:
> Send Nek5000-users mailing list submissions to
> nek5000-users at lists.mcs.anl.gov
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
> or, via email, send a message with subject or body 'help' to
> nek5000-users-request at lists.mcs.anl.gov
>
> You can reach the person managing the list at
> nek5000-users-owner at lists.mcs.anl.gov
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Nek5000-users digest..."
>
>
> Today's Topics:
>
> 1. Re: run-time hang up in gs_setup (nek5000-users at lists.mcs.anl.gov)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 12 Jan 2012 18:09:48 +0100
> From: nek5000-users at lists.mcs.anl.gov
> Subject: Re: [Nek5000-users] run-time hang up in gs_setup
> To: nek5000-users at lists.mcs.anl.gov
> Message-ID:
> <CAGTrLsaGQkuowxvfpK3AxvCtgCEkas9YBJkX024qF2Sh7RcmdA at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi Azad,
>
> can you try to run the turbChannel example using AMG and the latest
> version of the repo. Let me know if this works for you.
>
> Stefan
>
> On 1/11/12, nek5000-users at lists.mcs.anl.gov
> <nek5000-users at lists.mcs.anl.gov> wrote:
>> Dear Stefan and Aleks;
>>
>> Thanks for updating the wiki webpage regarding the AMG, although, I
>> persume there must be another step also there exist, namely: copy the
>> generated files from the amg_matlb to the running directory? (Or they
>> should be remained there and one puts the generated .dat files after
>> running the 3rd step?). By the way non of the versions I tried working
>> (even 707!) despite the fact that I had a range of matlab versions
>> tried. Hanging with the old version I had I compiled again and have
>> got the four files which was rather fast (with the message at the end:
>> Error contraction factor: 0.47...) I used them and every time during
>> the run-time it crashed simply:
>> ###########################################################################
>> AMG: reading through row 144540, pass 121/121
>> AMG: reading 0.071106 MB of W
>> AMG: reading 0.115601 MB of AfP
>> AMG: reading 0.132477 MB of Aff
>> AMG level 1 F-vars: 440159
>> AMG level 2 F-vars: 55146
>> AMG level 3 F-vars: 28480
>> AMG level 4 F-vars: 17051
>> AMG level 5 F-vars: 7524
>> AMG level 6 F-vars: 5711
>> AMG level 7 F-vars: 5763
>> AMG level 8 F-vars: 28583
>> AMG level 9 F-vars: 5380
>> AMG level 10 F-vars: 5737
>> Application 731033 exit codes: 139
>> Application 731033 exit signals: Killed
>> Application 731033 resources: utime ~417s, stime ~3s
>> ##########################################################################
>>
>> Can you help me with that cause I believe this case still doable with
>> correct AMG scheme.
>>
>> Many thanks
>> Azad
>>
>>
>>>
>>> Hi Azad,
>>>
>>> I believe old AMG files should work up to and including revision 707
>>> in case >you want to check AMG quickly.
>>>
>>> Best.
>>> Aleks
>>>
>>>
>>> ----- Original Message -----
>>> From: nek5000-users at lists.mcs.anl.gov
>>> To: nek5000-users at lists.mcs.anl.gov
>>> Sent: Tuesday, January 10, 2012 10:58:47 AM
>>> Subject: Re: [Nek5000-users] run-time hang up in gs_setup
>>>
>>> Hi Azad,
>>>
>>> your choice of lx1=8 is fine (it's our preferred sweet spot). If you
>>> have a large element count (say > 300'000) the factorization in the
>>> XXt setup phase may take hours. I guess that's why it looks like it's
>>> hanging. Again, there is a known bug which looks the same. So can't
>>> tell exactly what's causing your problem.
>>>
>>> I just updated the Wiki: https://nek5000.mcs.anl.gov/index.php/Amg_matlab
>>>
>>> Can you verify that it still fails.
>>>
>>> -Stefan
>>>
>>> Quoting nek5000-users-request at lists.mcs.anl.gov:
>>>
>>> Send Nek5000-users mailing list submissions to
>>> nek5000-users at lists.mcs.anl.gov
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>>> or, via email, send a message with subject or body 'help' to
>>> nek5000-users-request at lists.mcs.anl.gov
>>>
>>> You can reach the person managing the list at
>>> nek5000-users-owner at lists.mcs.anl.gov
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of Nek5000-users digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>> 1. Re: run-time hang up in gs_setup (nek5000-users at lists.mcs.anl.gov)
>>> 2. Re: run-time hang up in gs_setup (nek5000-users at lists.mcs.anl.gov)
>>> 3. Re: run-time hang up in gs_setup (nek5000-users at lists.mcs.anl.gov)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Tue, 10 Jan 2012 06:01:29 -0600 (CST)
>>> From: nek5000-users at lists.mcs.anl.gov
>>> Subject: Re: [Nek5000-users] run-time hang up in gs_setup
>>> To: nek5000-users at lists.mcs.anl.gov
>>> Message-ID: <Pine.LNX.4.64.1201100557250.6026 at v8.mcs.anl.gov>
>>> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
>>>
>>>
>>> Hi Azad,
>>>
>>> You are in record-setting territory for element counts! :)
>>>
>>> Are you using the amg-based coarse-grid solver?
>>> It is certain that you will need to do this (and,
>>> therefore, you will need matlab to process the AMG
>>> operators). There is some discussion of the steps
>>> on the wiki page. We can walk you through this process
>>> if you have any questions.
>>>
>>> What value of lx1 are you using?
>>>
>>> I would recommend fewer elements and a higher value of lx1.
>>> I think it will be easier to manage the data, etc.
>>>
>>> Paul
>>>
>>>
>>>
>>>
>>>
>>> On Tue, 10 Jan 2012, nek5000-users at lists.mcs.anl.gov wrote:
>>>
>>>> Dear NEKs;
>>>>
>>>> I am trying to run a simulation of a turbulent flow in a straight pipe
>>>> in high Reynolds number (Re_tau = 1000). After generating the grid with
>>>> PRENEK and extrude it using n2to3, the mesh ended up with 4,495,920
>>>> elements. It compiled properly; however, trying to run it, hanged up in
>>>> the last stage:
>>>> ########################################################################
>>>>
>>>> verify mesh topology
>>>> -1.000000000000000 1.000000000000000 Xrange
>>>> -1.000000000000000 1.000000000000000 Yrange
>>>> 0.000000000000000 25.00000000000000 Zrange
>>>> done :: verify mesh topology
>>>>
>>>> E-solver strategy: 1 itr
>>>> mg_nx: 1 3
>>>> mg_ny: 1 3
>>>> mg_nz: 1 3
>>>> call usrsetvert
>>>> done :: usrsetvert
>>>>
>>>> gs_setup: 866937 unique labels shared
>>>> pairwise times (avg, min, max): 0.000241442 0.00019722 0.000265908
>>>> crystal router : 0.000458177 0.000445795 0.000471807
>>>> used all_to_all method: pairwise
>>>> setupds time 5.6048E-02 seconds 1 2 4565612 4495920
>>>> setvert3d: 4 86046564 122013924 86046564 86046564
>>>> call usrsetvert
>>>> done :: usrsetvert
>>>>
>>>> gs_setup: 8041169 unique labels shared
>>>> pairwise times (avg, min, max): 0.00050716 0.000427103 0.00056479
>>>> crystal router : 0.0040165 0.00392921 0.00411811
>>>> used all_to_all method: pairwise
>>>> setupds time 1.0465E+00 seconds 2 4 86046564 4495920
>>>> setup h1 coarse grid, nx_crs= 2
>>>> call usrsetvert
>>>> done :: usrsetvert
>>>>
>>>> gs_setup: 866937 unique labels shared
>>>> pairwise times (avg, min, max): 0.000233683 0.000197816 0.00024941
>>>> crystal router : 0.000466869 0.00045588 0.000478101
>>>> used all_to_all method: pairwise
>>>> ########################################################################
>>>>
>>>>
>>>> I was wondering if you could help me with that. I attached the run
>>>> logfile and also genmap.out.
>>>>
>>>> Many thanks
>>>> Azad
>>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Tue, 10 Jan 2012 13:35:22 +0100
>>> From: nek5000-users at lists.mcs.anl.gov
>>> Subject: Re: [Nek5000-users] run-time hang up in gs_setup
>>> To: nek5000-users at lists.mcs.anl.gov
>>> Message-ID:
>>> <CAGTrLsaexkteQN1Y1NQ3FYz7Q2abb5YSLzOv+zeTwdvYXpD3Fw at mail.gmail.com>
>>> Content-Type: text/plain; charset=ISO-8859-1
>>>
>>> Hi Azad,
>>>
>>> We have seen similar situations. I think this has to do with a known
>>> bug. Unfortunately this bug is hard to reproduce and we haven't
>>> managed to fix it yet.
>>>
>>> -Stefan
>>>
>>> On 1/10/12, nek5000-users at lists.mcs.anl.gov
>>> <nek5000-users at lists.mcs.anl.gov> wrote:
>>>>
>>>> Hi Azad,
>>>>
>>>> You are in record-setting territory for element counts! :)
>>>>
>>>> Are you using the amg-based coarse-grid solver?
>>>> It is certain that you will need to do this (and,
>>>> therefore, you will need matlab to process the AMG
>>>> operators). There is some discussion of the steps
>>>> on the wiki page. We can walk you through this process
>>>> if you have any questions.
>>>>
>>>> What value of lx1 are you using?
>>>>
>>>> I would recommend fewer elements and a higher value of lx1.
>>>> I think it will be easier to manage the data, etc.
>>>>
>>>> Paul
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, 10 Jan 2012, nek5000-users at lists.mcs.anl.gov wrote:
>>>>
>>>>> Dear NEKs;
>>>>>
>>>>> I am trying to run a simulation of a turbulent flow in a straight pipe
>>>>> in high Reynolds number (Re_tau = 1000). After generating the grid with
>>>>> PRENEK and extrude it using n2to3, the mesh ended up with 4,495,920
>>>>> elements. It compiled properly; however, trying to run it, hanged up in
>>>>> the last stage:
>>>>> ########################################################################
>>>>>
>>>>> verify mesh topology
>>>>> -1.000000000000000 1.000000000000000 Xrange
>>>>> -1.000000000000000 1.000000000000000 Yrange
>>>>> 0.000000000000000 25.00000000000000 Zrange
>>>>> done :: verify mesh topology
>>>>>
>>>>> E-solver strategy: 1 itr
>>>>> mg_nx: 1 3
>>>>> mg_ny: 1 3
>>>>> mg_nz: 1 3
>>>>> call usrsetvert
>>>>> done :: usrsetvert
>>>>>
>>>>> gs_setup: 866937 unique labels shared
>>>>> pairwise times (avg, min, max): 0.000241442 0.00019722 0.000265908
>>>>> crystal router : 0.000458177 0.000445795 0.000471807
>>>>> used all_to_all method: pairwise
>>>>> setupds time 5.6048E-02 seconds 1 2 4565612 4495920
>>>>> setvert3d: 4 86046564 122013924 86046564 86046564
>>>>> call usrsetvert
>>>>> done :: usrsetvert
>>>>>
>>>>> gs_setup: 8041169 unique labels shared
>>>>> pairwise times (avg, min, max): 0.00050716 0.000427103 0.00056479
>>>>> crystal router : 0.0040165 0.00392921 0.00411811
>>>>> used all_to_all method: pairwise
>>>>> setupds time 1.0465E+00 seconds 2 4 86046564 4495920
>>>>> setup h1 coarse grid, nx_crs= 2
>>>>> call usrsetvert
>>>>> done :: usrsetvert
>>>>>
>>>>> gs_setup: 866937 unique labels shared
>>>>> pairwise times (avg, min, max): 0.000233683 0.000197816 0.00024941
>>>>> crystal router : 0.000466869 0.00045588 0.000478101
>>>>> used all_to_all method: pairwise
>>>>> ########################################################################
>>>>>
>>>>>
>>>>> I was wondering if you could help me with that. I attached the run
>>>>> logfile and also genmap.out.
>>>>>
>>>>> Many thanks
>>>>> Azad
>>>>>
>>>> _______________________________________________
>>>> Nek5000-users mailing list
>>>> Nek5000-users at lists.mcs.anl.gov
>>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 3
>>> Date: Tue, 10 Jan 2012 17:58:47 +0100
>>> From: nek5000-users at lists.mcs.anl.gov
>>> Subject: Re: [Nek5000-users] run-time hang up in gs_setup
>>> To: nek5000-users at lists.mcs.anl.gov
>>> Message-ID: <1326214727.2600.282.camel at damavand.mech.kth.se>
>>> Content-Type: text/plain; charset="UTF-8"
>>>
>>> Dear Paul and Stefan;
>>>
>>> Thanks very much for looking into it. I use polynomial order 7th
>>> (lx1=8). For the coarse-grid solver I actually used XXt. I also tried to
>>> use AMG, but unfortunately neither v619 nor the latest version could
>>> have compiled its matlab files and always gives me this error (in
>>> matlab/R2011a):
>>> ##############################################
>>> ...
>>> sparsification tolerance [1e-4]: stol = 0.0001
>>>
>>> ------------------------------------------------------------------------
>>> Segmentation violation detected at Tue Jan 10 15:56:46 2012
>>> ------------------------------------------------------------------------
>>> ....
>>> Abnormal termination:
>>> Segmentation violation
>>> ....
>>> #############################################
>>> I have been in the web page: "amg_matlab Matlab based tool to generate
>>> AMG solver inputfiles" (http://nek5000.mcs.anl.gov/index.php/Amg_matlab)
>>> which gives me an empty link.
>>>
>>> I had an old version of the .dat files needed to run AMG, which I tried
>>> those as (amg_Aff.dat, amgdmp_i.dat, amg.dat, amg_AfP.dat, amgdmp_p.dat,
>>> amgdmp_j.dat, amg_W.dat) and I have got this error:
>>>
>>> ############################################
>>> ...
>>> AMG: reading through row 142800, pass 119/121
>>> AMG: reading through row 144000, pass 120/121
>>> AMG: reading through row 144540, pass 121/121
>>> ERROR (proc
>>> 0000,
>>> /afs/pdc.kth.se/home/a/anoorani/codes/latest_nek/nek5_svn/trunk/nek/jl/amg.c:468):
>>> AMG: missing data for some
>>> rows
>>>
>>> call exitt: dying ...
>>> ############################################
>>>
>>> I think AMG could be a possibility to overcome this problem, though I
>>> could not manage to get a run with that one. I look into the problem
>>> with higher polynomial order to see if it reduces the number of elements
>>> dramatically, or at least resolve this issue.
>>>
>>> Best regards
>>> Azad
>>>
>>> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
>>>> Hi Azad,
>>>>
>>>> We have seen similar situations. I think this has to do with a known
>>>> bug. Unfortunately this bug is hard to reproduce and we haven't
>>>> managed to fix it yet.
>>>>
>>>> -Stefan
>>>>
>>>> On 1/10/12, nek5000-users at lists.mcs.anl.gov
>>>> <nek5000-users at lists.mcs.anl.gov> wrote:
>>>>
>>>>
>>>> Hi Azad,
>>>>
>>>> You are in record-setting territory for element counts! :)
>>>>
>>>> Are you using the amg-based coarse-grid solver?
>>>> It is certain that you will need to do this (and,
>>>> therefore, you will need matlab to process the AMG
>>>> operators). There is some discussion of the steps
>>>> on the wiki page. We can walk you through this process
>>>> if you have any questions.
>>>>
>>>> What value of lx1 are you using?
>>>>
>>>> I would recommend fewer elements and a higher value of lx1.
>>>> I think it will be easier to manage the data, etc.
>>>>
>>>> Paul
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, 10 Jan 2012, nek5000-users at lists.mcs.anl.gov wrote:
>>>>
>>>> Dear NEKs;
>>>>
>>>> I am trying to run a simulation of a turbulent flow in a straight pipe
>>>> in high Reynolds number (Re_tau = 1000). After generating the grid with
>>>> PRENEK and extrude it using n2to3, the mesh ended up with 4,495,920
>>>> elements. It compiled properly; however, trying to run it, hanged up in
>>>> the last stage:
>>>> ########################################################################
>>>>
>>>> verify mesh topology
>>>> -1.000000000000000 1.000000000000000 Xrange
>>>> -1.000000000000000 1.000000000000000 Yrange
>>>> 0.000000000000000 25.00000000000000 Zrange
>>>> done :: verify mesh topology
>>>>
>>>> E-solver strategy: 1 itr
>>>> mg_nx: 1 3
>>>> mg_ny: 1 3
>>>> mg_nz: 1 3
>>>> call usrsetvert
>>>> done :: usrsetvert
>>>>
>>>> gs_setup: 866937 unique labels shared
>>>> pairwise times (avg, min, max): 0.000241442 0.00019722 0.000265908
>>>> crystal router : 0.000458177 0.000445795 0.000471807
>>>> used all_to_all method: pairwise
>>>> setupds time 5.6048E-02 seconds 1 2 4565612 4495920
>>>> setvert3d: 4 86046564 122013924 86046564 86046564
>>>> call usrsetvert
>>>> done :: usrsetvert
>>>>
>>>> gs_setup: 8041169 unique labels shared
>>>> pairwise times (avg, min, max): 0.00050716 0.000427103 0.00056479
>>>> crystal router : 0.0040165 0.00392921 0.00411811
>>>> used all_to_all method: pairwise
>>>> setupds time 1.0465E+00 seconds 2 4 86046564 4495920
>>>> setup h1 coarse grid, nx_crs= 2
>>>> call usrsetvert
>>>> done :: usrsetvert
>>>>
>>>> gs_setup: 866937 unique labels shared
>>>> pairwise times (avg, min, max): 0.000233683 0.000197816 0.00024941
>>>> crystal router : 0.000466869 0.00045588 0.000478101
>>>> used all_to_all method: pairwise
>>>> ########################################################################
>>>>
>>>>
>>>> I was wondering if you could help me with that. I attached the run
>>>> logfile and also genmap.out.
>>>>
>>>> Many thanks
>>>> Azad
>>>>
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> Nek5000-users mailing list
>>> Nek5000-users at lists.mcs.anl.gov
>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>>>
>>>
>>> End of Nek5000-users Digest, Vol 35, Issue 5
>>> ********************************************
>>>
>>
>>
>>
>> ----------------------------------------------------------------
>> This message was sent using IMP, the Internet Messaging Program.
>> _______________________________________________
>> Nek5000-users mailing list
>> Nek5000-users at lists.mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>>
>
>
> ------------------------------
>
> _______________________________________________
> Nek5000-users mailing list
> Nek5000-users at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>
>
> End of Nek5000-users Digest, Vol 35, Issue 7
> ********************************************
>
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
More information about the Nek5000-users
mailing list