[Nek5000-users] run-time hang up in gs_setup
nek5000-users at lists.mcs.anl.gov
nek5000-users at lists.mcs.anl.gov
Thu Jan 12 14:09:12 CST 2012
Hi Azad,
Earlier you had asked if a step was missing in the wiki with regards to the
amg files/scripts. The has just been updated.
Can you tell me exactly what you did to run the turbChannel test? And what
output you had from the amg_matlab/run step?
thanks,
Katie
On Thu, Jan 12, 2012 at 1:38 PM, <nek5000-users at lists.mcs.anl.gov> wrote:
> Hi Stefan;
>
> Unfortunately it does not work. first of all, ./run in amg_matlab does not
> produce anything (latest version). Using my version of the files (running
> turbChannel), the simulation crashed giving me this error:
>
> ##############################**#################
> AMG level 8: 3 iterations with rho = 0.680429
> AMG level 9: 2 iterations with rho = 0.560188
> AMG: 144540 rows
> AMG: reading through row 1200, pass 1/121
> ERROR (proc 0000, /scratch/azad/codes/late/nek5_**svn/trunk/nek/jl/amg.c:875):
> AMG: data
> has more rows than given problem
>
> call exitt: dying ...
> ##############################**#################
>
> Regards
> Azad
>
> Quoting nek5000-users-request at lists.**mcs.anl.gov<nek5000-users-request at lists.mcs.anl.gov>
> :
>
> Send Nek5000-users mailing list submissions to
>> nek5000-users at lists.mcs.anl.**gov<nek5000-users at lists.mcs.anl.gov>
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://lists.mcs.anl.gov/**mailman/listinfo/nek5000-users<https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users>
>> or, via email, send a message with subject or body 'help' to
>> nek5000-users-request at lists.**mcs.anl.gov<nek5000-users-request at lists.mcs.anl.gov>
>>
>> You can reach the person managing the list at
>> nek5000-users-owner at lists.mcs.**anl.gov<nek5000-users-owner at lists.mcs.anl.gov>
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Nek5000-users digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: run-time hang up in gs_setup (nek5000-users at lists.mcs.anl.**gov<nek5000-users at lists.mcs.anl.gov>
>> )
>>
>>
>> ------------------------------**------------------------------**
>> ----------
>>
>> Message: 1
>> Date: Thu, 12 Jan 2012 18:09:48 +0100
>>
>> From: nek5000-users at lists.mcs.anl.**gov <nek5000-users at lists.mcs.anl.gov>
>> Subject: Re: [Nek5000-users] run-time hang up in gs_setup
>> To: nek5000-users at lists.mcs.anl.**gov <nek5000-users at lists.mcs.anl.gov>
>> Message-ID:
>> <CAGTrLsaGQkuowxvfpK3AxvCtgCEk**as9YBJkX024qF2Sh7RcmdA at mail.**
>> gmail.com<CAGTrLsaGQkuowxvfpK3AxvCtgCEkas9YBJkX024qF2Sh7RcmdA at mail.gmail.com>
>> >
>>
>> Content-Type: text/plain; charset=ISO-8859-1
>>
>> Hi Azad,
>>
>> can you try to run the turbChannel example using AMG and the latest
>> version of the repo. Let me know if this works for you.
>>
>> Stefan
>>
>> On 1/11/12, nek5000-users at lists.mcs.anl.**gov<nek5000-users at lists.mcs.anl.gov>
>> <nek5000-users at lists.mcs.anl.**gov <nek5000-users at lists.mcs.anl.gov>>
>> wrote:
>>
>>> Dear Stefan and Aleks;
>>>
>>> Thanks for updating the wiki webpage regarding the AMG, although, I
>>> persume there must be another step also there exist, namely: copy the
>>> generated files from the amg_matlb to the running directory? (Or they
>>> should be remained there and one puts the generated .dat files after
>>> running the 3rd step?). By the way non of the versions I tried working
>>> (even 707!) despite the fact that I had a range of matlab versions
>>> tried. Hanging with the old version I had I compiled again and have
>>> got the four files which was rather fast (with the message at the end:
>>> Error contraction factor: 0.47...) I used them and every time during
>>> the run-time it crashed simply:
>>> ##############################**##############################**
>>> ###############
>>> AMG: reading through row 144540, pass 121/121
>>> AMG: reading 0.071106 MB of W
>>> AMG: reading 0.115601 MB of AfP
>>> AMG: reading 0.132477 MB of Aff
>>> AMG level 1 F-vars: 440159
>>> AMG level 2 F-vars: 55146
>>> AMG level 3 F-vars: 28480
>>> AMG level 4 F-vars: 17051
>>> AMG level 5 F-vars: 7524
>>> AMG level 6 F-vars: 5711
>>> AMG level 7 F-vars: 5763
>>> AMG level 8 F-vars: 28583
>>> AMG level 9 F-vars: 5380
>>> AMG level 10 F-vars: 5737
>>> Application 731033 exit codes: 139
>>> Application 731033 exit signals: Killed
>>> Application 731033 resources: utime ~417s, stime ~3s
>>> ##############################**##############################**
>>> ##############
>>>
>>> Can you help me with that cause I believe this case still doable with
>>> correct AMG scheme.
>>>
>>> Many thanks
>>> Azad
>>>
>>>
>>>
>>>> Hi Azad,
>>>>
>>>> I believe old AMG files should work up to and including revision 707
>>>> in case >you want to check AMG quickly.
>>>>
>>>> Best.
>>>> Aleks
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: nek5000-users at lists.mcs.anl.gov
>>>> To: nek5000-users at lists.mcs.anl.gov
>>>> Sent: Tuesday, January 10, 2012 10:58:47 AM
>>>> Subject: Re: [Nek5000-users] run-time hang up in gs_setup
>>>>
>>>> Hi Azad,
>>>>
>>>> your choice of lx1=8 is fine (it's our preferred sweet spot). If you
>>>> have a large element count (say > 300'000) the factorization in the
>>>> XXt setup phase may take hours. I guess that's why it looks like it's
>>>> hanging. Again, there is a known bug which looks the same. So can't
>>>> tell exactly what's causing your problem.
>>>>
>>>> I just updated the Wiki: https://nek5000.mcs.anl.gov/**
>>>> index.php/Amg_matlab <https://nek5000.mcs.anl.gov/index.php/Amg_matlab>
>>>>
>>>> Can you verify that it still fails.
>>>>
>>>> -Stefan
>>>>
>>>> Quoting nek5000-users-request at lists.**mcs.anl.gov<nek5000-users-request at lists.mcs.anl.gov>
>>>> :
>>>>
>>>> Send Nek5000-users mailing list submissions to
>>>> nek5000-users at lists.mcs.anl.**gov<nek5000-users at lists.mcs.anl.gov>
>>>>
>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>> https://lists.mcs.anl.gov/**mailman/listinfo/nek5000-users<https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users>
>>>> or, via email, send a message with subject or body 'help' to
>>>> nek5000-users-request at lists.**mcs.anl.gov<nek5000-users-request at lists.mcs.anl.gov>
>>>>
>>>> You can reach the person managing the list at
>>>> nek5000-users-owner at lists.mcs.**anl.gov<nek5000-users-owner at lists.mcs.anl.gov>
>>>>
>>>> When replying, please edit your Subject line so it is more specific
>>>> than "Re: Contents of Nek5000-users digest..."
>>>>
>>>>
>>>> Today's Topics:
>>>>
>>>> 1. Re: run-time hang up in gs_setup (nek5000-users at lists.mcs.anl.**
>>>> gov <nek5000-users at lists.mcs.anl.gov>)
>>>> 2. Re: run-time hang up in gs_setup (nek5000-users at lists.mcs.anl.**
>>>> gov <nek5000-users at lists.mcs.anl.gov>)
>>>> 3. Re: run-time hang up in gs_setup (nek5000-users at lists.mcs.anl.**
>>>> gov <nek5000-users at lists.mcs.anl.gov>)
>>>>
>>>>
>>>> ------------------------------**------------------------------**
>>>> ----------
>>>>
>>>> Message: 1
>>>> Date: Tue, 10 Jan 2012 06:01:29 -0600 (CST)
>>>> From: nek5000-users at lists.mcs.anl.**gov<nek5000-users at lists.mcs.anl.gov>
>>>> Subject: Re: [Nek5000-users] run-time hang up in gs_setup
>>>> To: nek5000-users at lists.mcs.anl.**gov <nek5000-users at lists.mcs.anl.gov>
>>>> Message-ID: <Pine.LNX.4.64.1201100557250.**6026 at v8.mcs.anl.gov<Pine.LNX.4.64.1201100557250.6026 at v8.mcs.anl.gov>
>>>> >
>>>> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
>>>>
>>>>
>>>> Hi Azad,
>>>>
>>>> You are in record-setting territory for element counts! :)
>>>>
>>>> Are you using the amg-based coarse-grid solver?
>>>> It is certain that you will need to do this (and,
>>>> therefore, you will need matlab to process the AMG
>>>> operators). There is some discussion of the steps
>>>> on the wiki page. We can walk you through this process
>>>> if you have any questions.
>>>>
>>>> What value of lx1 are you using?
>>>>
>>>> I would recommend fewer elements and a higher value of lx1.
>>>> I think it will be easier to manage the data, etc.
>>>>
>>>> Paul
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, 10 Jan 2012, nek5000-users at lists.mcs.anl.**gov<nek5000-users at lists.mcs.anl.gov>wrote:
>>>>
>>>> Dear NEKs;
>>>>>
>>>>> I am trying to run a simulation of a turbulent flow in a straight pipe
>>>>> in high Reynolds number (Re_tau = 1000). After generating the grid with
>>>>> PRENEK and extrude it using n2to3, the mesh ended up with 4,495,920
>>>>> elements. It compiled properly; however, trying to run it, hanged up in
>>>>> the last stage:
>>>>> ##############################**##############################**
>>>>> ############
>>>>>
>>>>> verify mesh topology
>>>>> -1.000000000000000 1.000000000000000 Xrange
>>>>> -1.000000000000000 1.000000000000000 Yrange
>>>>> 0.000000000000000 25.00000000000000 Zrange
>>>>> done :: verify mesh topology
>>>>>
>>>>> E-solver strategy: 1 itr
>>>>> mg_nx: 1 3
>>>>> mg_ny: 1 3
>>>>> mg_nz: 1 3
>>>>> call usrsetvert
>>>>> done :: usrsetvert
>>>>>
>>>>> gs_setup: 866937 unique labels shared
>>>>> pairwise times (avg, min, max): 0.000241442 0.00019722 0.000265908
>>>>> crystal router : 0.000458177 0.000445795 0.000471807
>>>>> used all_to_all method: pairwise
>>>>> setupds time 5.6048E-02 seconds 1 2 4565612 4495920
>>>>> setvert3d: 4 86046564 122013924 86046564 86046564
>>>>> call usrsetvert
>>>>> done :: usrsetvert
>>>>>
>>>>> gs_setup: 8041169 unique labels shared
>>>>> pairwise times (avg, min, max): 0.00050716 0.000427103 0.00056479
>>>>> crystal router : 0.0040165 0.00392921 0.00411811
>>>>> used all_to_all method: pairwise
>>>>> setupds time 1.0465E+00 seconds 2 4 86046564 4495920
>>>>> setup h1 coarse grid, nx_crs= 2
>>>>> call usrsetvert
>>>>> done :: usrsetvert
>>>>>
>>>>> gs_setup: 866937 unique labels shared
>>>>> pairwise times (avg, min, max): 0.000233683 0.000197816 0.00024941
>>>>> crystal router : 0.000466869 0.00045588 0.000478101
>>>>> used all_to_all method: pairwise
>>>>> ##############################**##############################**
>>>>> ############
>>>>>
>>>>>
>>>>> I was wondering if you could help me with that. I attached the run
>>>>> logfile and also genmap.out.
>>>>>
>>>>> Many thanks
>>>>> Azad
>>>>>
>>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 2
>>>> Date: Tue, 10 Jan 2012 13:35:22 +0100
>>>> From: nek5000-users at lists.mcs.anl.**gov<nek5000-users at lists.mcs.anl.gov>
>>>> Subject: Re: [Nek5000-users] run-time hang up in gs_setup
>>>> To: nek5000-users at lists.mcs.anl.**gov <nek5000-users at lists.mcs.anl.gov>
>>>> Message-ID:
>>>> <CAGTrLsaexkteQN1Y1NQ3FYz7Q2ab**b5YSLzOv+zeTwdvYXpD3Fw at mail.**
>>>> gmail.com<CAGTrLsaexkteQN1Y1NQ3FYz7Q2abb5YSLzOv%2BzeTwdvYXpD3Fw at mail.gmail.com>
>>>> >
>>>> Content-Type: text/plain; charset=ISO-8859-1
>>>>
>>>> Hi Azad,
>>>>
>>>> We have seen similar situations. I think this has to do with a known
>>>> bug. Unfortunately this bug is hard to reproduce and we haven't
>>>> managed to fix it yet.
>>>>
>>>> -Stefan
>>>>
>>>> On 1/10/12, nek5000-users at lists.mcs.anl.**gov<nek5000-users at lists.mcs.anl.gov>
>>>> <nek5000-users at lists.mcs.anl.**gov <nek5000-users at lists.mcs.anl.gov>>
>>>> wrote:
>>>>
>>>>>
>>>>> Hi Azad,
>>>>>
>>>>> You are in record-setting territory for element counts! :)
>>>>>
>>>>> Are you using the amg-based coarse-grid solver?
>>>>> It is certain that you will need to do this (and,
>>>>> therefore, you will need matlab to process the AMG
>>>>> operators). There is some discussion of the steps
>>>>> on the wiki page. We can walk you through this process
>>>>> if you have any questions.
>>>>>
>>>>> What value of lx1 are you using?
>>>>>
>>>>> I would recommend fewer elements and a higher value of lx1.
>>>>> I think it will be easier to manage the data, etc.
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, 10 Jan 2012, nek5000-users at lists.mcs.anl.**gov<nek5000-users at lists.mcs.anl.gov>wrote:
>>>>>
>>>>> Dear NEKs;
>>>>>>
>>>>>> I am trying to run a simulation of a turbulent flow in a straight pipe
>>>>>> in high Reynolds number (Re_tau = 1000). After generating the grid
>>>>>> with
>>>>>> PRENEK and extrude it using n2to3, the mesh ended up with 4,495,920
>>>>>> elements. It compiled properly; however, trying to run it, hanged up
>>>>>> in
>>>>>> the last stage:
>>>>>> ##############################**##############################**
>>>>>> ############
>>>>>>
>>>>>> verify mesh topology
>>>>>> -1.000000000000000 1.000000000000000 Xrange
>>>>>> -1.000000000000000 1.000000000000000 Yrange
>>>>>> 0.000000000000000 25.00000000000000 Zrange
>>>>>> done :: verify mesh topology
>>>>>>
>>>>>> E-solver strategy: 1 itr
>>>>>> mg_nx: 1 3
>>>>>> mg_ny: 1 3
>>>>>> mg_nz: 1 3
>>>>>> call usrsetvert
>>>>>> done :: usrsetvert
>>>>>>
>>>>>> gs_setup: 866937 unique labels shared
>>>>>> pairwise times (avg, min, max): 0.000241442 0.00019722 0.000265908
>>>>>> crystal router : 0.000458177 0.000445795 0.000471807
>>>>>> used all_to_all method: pairwise
>>>>>> setupds time 5.6048E-02 seconds 1 2 4565612 4495920
>>>>>> setvert3d: 4 86046564 122013924 86046564 86046564
>>>>>> call usrsetvert
>>>>>> done :: usrsetvert
>>>>>>
>>>>>> gs_setup: 8041169 unique labels shared
>>>>>> pairwise times (avg, min, max): 0.00050716 0.000427103 0.00056479
>>>>>> crystal router : 0.0040165 0.00392921 0.00411811
>>>>>> used all_to_all method: pairwise
>>>>>> setupds time 1.0465E+00 seconds 2 4 86046564 4495920
>>>>>> setup h1 coarse grid, nx_crs= 2
>>>>>> call usrsetvert
>>>>>> done :: usrsetvert
>>>>>>
>>>>>> gs_setup: 866937 unique labels shared
>>>>>> pairwise times (avg, min, max): 0.000233683 0.000197816 0.00024941
>>>>>> crystal router : 0.000466869 0.00045588 0.000478101
>>>>>> used all_to_all method: pairwise
>>>>>> ##############################**##############################**
>>>>>> ############
>>>>>>
>>>>>>
>>>>>> I was wondering if you could help me with that. I attached the run
>>>>>> logfile and also genmap.out.
>>>>>>
>>>>>> Many thanks
>>>>>> Azad
>>>>>>
>>>>>> ______________________________**_________________
>>>>> Nek5000-users mailing list
>>>>> Nek5000-users at lists.mcs.anl.**gov <Nek5000-users at lists.mcs.anl.gov>
>>>>> https://lists.mcs.anl.gov/**mailman/listinfo/nek5000-users<https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users>
>>>>>
>>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 3
>>>> Date: Tue, 10 Jan 2012 17:58:47 +0100
>>>> From: nek5000-users at lists.mcs.anl.**gov<nek5000-users at lists.mcs.anl.gov>
>>>> Subject: Re: [Nek5000-users] run-time hang up in gs_setup
>>>> To: nek5000-users at lists.mcs.anl.**gov <nek5000-users at lists.mcs.anl.gov>
>>>> Message-ID: <1326214727.2600.282.camel@**damavand.mech.kth.se<1326214727.2600.282.camel at damavand.mech.kth.se>
>>>> >
>>>> Content-Type: text/plain; charset="UTF-8"
>>>>
>>>> Dear Paul and Stefan;
>>>>
>>>> Thanks very much for looking into it. I use polynomial order 7th
>>>> (lx1=8). For the coarse-grid solver I actually used XXt. I also tried to
>>>> use AMG, but unfortunately neither v619 nor the latest version could
>>>> have compiled its matlab files and always gives me this error (in
>>>> matlab/R2011a):
>>>> ##############################**################
>>>> ...
>>>> sparsification tolerance [1e-4]: stol = 0.0001
>>>>
>>>> ------------------------------**------------------------------**
>>>> ------------
>>>> Segmentation violation detected at Tue Jan 10 15:56:46 2012
>>>> ------------------------------**------------------------------**
>>>> ------------
>>>> ....
>>>> Abnormal termination:
>>>> Segmentation violation
>>>> ....
>>>> ##############################**###############
>>>> I have been in the web page: "amg_matlab Matlab based tool to generate
>>>> AMG solver inputfiles" (http://nek5000.mcs.anl.gov/**
>>>> index.php/Amg_matlab <http://nek5000.mcs.anl.gov/index.php/Amg_matlab>)
>>>> which gives me an empty link.
>>>>
>>>> I had an old version of the .dat files needed to run AMG, which I tried
>>>> those as (amg_Aff.dat, amgdmp_i.dat, amg.dat, amg_AfP.dat, amgdmp_p.dat,
>>>> amgdmp_j.dat, amg_W.dat) and I have got this error:
>>>>
>>>> ##############################**##############
>>>> ...
>>>> AMG: reading through row 142800, pass 119/121
>>>> AMG: reading through row 144000, pass 120/121
>>>> AMG: reading through row 144540, pass 121/121
>>>> ERROR (proc
>>>> 0000,
>>>> /afs/pdc.kth.se/home/a/**anoorani/codes/latest_nek/**
>>>> nek5_svn/trunk/nek/jl/amg.c:**468<http://pdc.kth.se/home/a/anoorani/codes/latest_nek/nek5_svn/trunk/nek/jl/amg.c:468>
>>>> ):
>>>> AMG: missing data for some
>>>> rows
>>>>
>>>> call exitt: dying ...
>>>> ##############################**##############
>>>>
>>>> I think AMG could be a possibility to overcome this problem, though I
>>>> could not manage to get a run with that one. I look into the problem
>>>> with higher polynomial order to see if it reduces the number of elements
>>>> dramatically, or at least resolve this issue.
>>>>
>>>> Best regards
>>>> Azad
>>>>
>>>> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%**%%%%%%%%%%%%%%%%%%%%%%%%%
>>>>
>>>>> Hi Azad,
>>>>>
>>>>> We have seen similar situations. I think this has to do with a known
>>>>> bug. Unfortunately this bug is hard to reproduce and we haven't
>>>>> managed to fix it yet.
>>>>>
>>>>> -Stefan
>>>>>
>>>>> On 1/10/12, nek5000-users at lists.mcs.anl.gov
>>>>> <nek5000-users at lists.mcs.anl.gov> wrote:
>>>>>
>>>>>
>>>>> Hi Azad,
>>>>>
>>>>> You are in record-setting territory for element counts! :)
>>>>>
>>>>> Are you using the amg-based coarse-grid solver?
>>>>> It is certain that you will need to do this (and,
>>>>> therefore, you will need matlab to process the AMG
>>>>> operators). There is some discussion of the steps
>>>>> on the wiki page. We can walk you through this process
>>>>> if you have any questions.
>>>>>
>>>>> What value of lx1 are you using?
>>>>>
>>>>> I would recommend fewer elements and a higher value of lx1.
>>>>> I think it will be easier to manage the data, etc.
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, 10 Jan 2012, nek5000-users at lists.mcs.anl.gov wrote:
>>>>>
>>>>> Dear NEKs;
>>>>>
>>>>> I am trying to run a simulation of a turbulent flow in a straight pipe
>>>>> in high Reynolds number (Re_tau = 1000). After generating the grid with
>>>>> PRENEK and extrude it using n2to3, the mesh ended up with 4,495,920
>>>>> elements. It compiled properly; however, trying to run it, hanged up in
>>>>> the last stage:
>>>>> ##############################**##############################**
>>>>> ############
>>>>>
>>>>> verify mesh topology
>>>>> -1.000000000000000 1.000000000000000 Xrange
>>>>> -1.000000000000000 1.000000000000000 Yrange
>>>>> 0.000000000000000 25.00000000000000 Zrange
>>>>> done :: verify mesh topology
>>>>>
>>>>> E-solver strategy: 1 itr
>>>>> mg_nx: 1 3
>>>>> mg_ny: 1 3
>>>>> mg_nz: 1 3
>>>>> call usrsetvert
>>>>> done :: usrsetvert
>>>>>
>>>>> gs_setup: 866937 unique labels shared
>>>>> pairwise times (avg, min, max): 0.000241442 0.00019722 0.000265908
>>>>> crystal router : 0.000458177 0.000445795 0.000471807
>>>>> used all_to_all method: pairwise
>>>>> setupds time 5.6048E-02 seconds 1 2 4565612 4495920
>>>>> setvert3d: 4 86046564 122013924 86046564 86046564
>>>>> call usrsetvert
>>>>> done :: usrsetvert
>>>>>
>>>>> gs_setup: 8041169 unique labels shared
>>>>> pairwise times (avg, min, max): 0.00050716 0.000427103 0.00056479
>>>>> crystal router : 0.0040165 0.00392921 0.00411811
>>>>> used all_to_all method: pairwise
>>>>> setupds time 1.0465E+00 seconds 2 4 86046564 4495920
>>>>> setup h1 coarse grid, nx_crs= 2
>>>>> call usrsetvert
>>>>> done :: usrsetvert
>>>>>
>>>>> gs_setup: 866937 unique labels shared
>>>>> pairwise times (avg, min, max): 0.000233683 0.000197816 0.00024941
>>>>> crystal router : 0.000466869 0.00045588 0.000478101
>>>>> used all_to_all method: pairwise
>>>>> ##############################**##############################**
>>>>> ############
>>>>>
>>>>>
>>>>> I was wondering if you could help me with that. I attached the run
>>>>> logfile and also genmap.out.
>>>>>
>>>>> Many thanks
>>>>> Azad
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> ______________________________**_________________
>>>> Nek5000-users mailing list
>>>> Nek5000-users at lists.mcs.anl.**gov <Nek5000-users at lists.mcs.anl.gov>
>>>> https://lists.mcs.anl.gov/**mailman/listinfo/nek5000-users<https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users>
>>>>
>>>>
>>>> End of Nek5000-users Digest, Vol 35, Issue 5
>>>> **********************************************
>>>>
>>>>
>>>
>>>
>>> ------------------------------**------------------------------**----
>>> This message was sent using IMP, the Internet Messaging Program.
>>> ______________________________**_________________
>>> Nek5000-users mailing list
>>> Nek5000-users at lists.mcs.anl.**gov <Nek5000-users at lists.mcs.anl.gov>
>>> https://lists.mcs.anl.gov/**mailman/listinfo/nek5000-users<https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users>
>>>
>>>
>>
>> ------------------------------
>>
>> ______________________________**_________________
>> Nek5000-users mailing list
>> Nek5000-users at lists.mcs.anl.**gov <Nek5000-users at lists.mcs.anl.gov>
>> https://lists.mcs.anl.gov/**mailman/listinfo/nek5000-users<https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users>
>>
>>
>> End of Nek5000-users Digest, Vol 35, Issue 7
>> **********************************************
>>
>>
>
>
> ------------------------------**------------------------------**----
> This message was sent using IMP, the Internet Messaging Program.
> ______________________________**_________________
> Nek5000-users mailing list
> Nek5000-users at lists.mcs.anl.**gov <Nek5000-users at lists.mcs.anl.gov>
> https://lists.mcs.anl.gov/**mailman/listinfo/nek5000-users<https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/nek5000-users/attachments/20120112/b0da596a/attachment.html>
More information about the Nek5000-users
mailing list