From nbest at ci.uchicago.edu  Mon Dec  3 18:07:17 2012
From: nbest at ci.uchicago.edu (Neil Best)
Date: Mon, 3 Dec 2012 18:07:17 -0600
Subject: [Swift-user] swift on midway: apps and modules
In-Reply-To: <364676584.130715.1354230265164.JavaMail.root@zimbra-mb2.anl.gov>
References: <CANVU8H19QVCmfBsDqAf6djmfsyPXiMBLSO8GBy-JSNtmuN1Qjg@mail.gmail.com>
	<364676584.130715.1354230265164.JavaMail.root@zimbra-mb2.anl.gov>
Message-ID: <CANVU8H3om0Gizxdh185gAX63+hWJ89CzqJ=pJiiz3KZ6TVD6xQ@mail.gmail.com>

On Thu, Nov 29, 2012 at 5:04 PM, David Kelly <davidk at ci.uchicago.edu> wrote:
> Hi Neil,
>
> It could possibly be something environmental. Are the modules loaded automatically when you log in through your .bashrc? If they are not already, could you give that a try?
>
> If that doesn't work, try editing sites.xml and try changing
>
> <workdirectory>/scratch/local/swift</workdirectory>
> to
> <workdirectory>/project/joshuaelliott/narr</workdirectory>
>
> Then try the run again and let me know when it's finished. That will just make some extra debugging information available that might better explain why it's failing.
>
> Thanks,
> David
>


Everything was basically working except that my project directory
became too cluttered with temp files so I changed the workdirectory
back to /scratch/... and now swift is failing with that "exit code
127" message again.  What's the correlation between workdirectory and
module/PATH problems?  Or is there something else going on here?


From wilde at mcs.anl.gov  Mon Dec  3 18:22:26 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Mon, 3 Dec 2012 18:22:26 -0600 (CST)
Subject: [Swift-user] swift on midway: apps and modules
In-Reply-To: <CANVU8H3om0Gizxdh185gAX63+hWJ89CzqJ=pJiiz3KZ6TVD6xQ@mail.gmail.com>
Message-ID: <1415331725.68060.1354580546585.JavaMail.root@zimbra.anl.gov>

Instead of /scratch/local try /scratch/midway/nbest

I suspect the error here is that you are setting the workdirectory to  path that is not shared between the login host and compute nodes.

If /scratch/midway/nbest works (and thus confirms this suspicion), and you still want to place your work directory on a node-local filesystem, you can try to turn on "coaster staging".  To do that, set the work directory back to /scratch/local, and in your properties file (in the "cf" file generated by the bin/runswift command in the RCC tutorial examples) change:

  use.provider.staging=true

to

  use.provider.staging=true

Now your input data files will be copied to the node /scratch/local dir and your output files copied back.

I just tested this using the workdirectory /scratch/local/wilde/swiftwork, which seemed to work, but I did not check everything carefully yet.

- Mike


----- Original Message -----
> From: "Neil Best" <nbest at ci.uchicago.edu>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: swift-user at ci.uchicago.edu
> Sent: Monday, December 3, 2012 6:07:17 PM
> Subject: Re: [Swift-user] swift on midway: apps and modules
> On Thu, Nov 29, 2012 at 5:04 PM, David Kelly <davidk at ci.uchicago.edu>
> wrote:
> > Hi Neil,
> >
> > It could possibly be something environmental. Are the modules loaded
> > automatically when you log in through your .bashrc? If they are not
> > already, could you give that a try?
> >
> > If that doesn't work, try editing sites.xml and try changing
> >
> > <workdirectory>/scratch/local/swift</workdirectory>
> > to
> > <workdirectory>/project/joshuaelliott/narr</workdirectory>
> >
> > Then try the run again and let me know when it's finished. That will
> > just make some extra debugging information available that might
> > better explain why it's failing.
> >
> > Thanks,
> > David
> >
> 
> 
> Everything was basically working except that my project directory
> became too cluttered with temp files so I changed the workdirectory
> back to /scratch/... and now swift is failing with that "exit code
> 127" message again. What's the correlation between workdirectory and
> module/PATH problems? Or is there something else going on here?
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From lpesce at uchicago.edu  Tue Dec  4 19:53:35 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Tue, 4 Dec 2012 19:53:35 -0600
Subject: [Swift-user] 0.94-2012.1102 Error:
	org.globus.cog.karajan.translator.atoms.SingleLineComment$Eval
	cannot be cast to
	org.globus.cog.karajan.translator.atoms.Identifier$Eval
References: <20121205014215.A074F9D231@vm-125-55.ci.uchicago.edu>
Message-ID: <C3C93D8A-E48F-4D71-AFB5-BA8BCB03AB46@uchicago.edu>

Had anyone ever seen this error?


Any ideas? 

SWIFT_HOME=/soft/swift/0.94-2012.1102

ERROR:
Could not start execution:
      Error reading source: org.globus.cog.karajan.parser.ParsingException: Exception executing org.globus.cog.karajan.translator.atoms.Named at 2cca7081:
      org.globus.cog.karajan.parser.ParsingException: Exception executing org.globus.cog.karajan.translator.atoms.Named at 2cca7081:
      Exception executing org.globus.cog.karajan.translator.atoms.Named at 2cca7081:
      Exception executing org.globus.cog.karajan.translator.atoms.Named at 2cca7081:
      org.globus.cog.karajan.translator.atoms.SingleLineComment$Eval cannot be cast to org.globus.cog.karajan.translator.atoms.Identifier$Eval

type file;

string MCRPath = "/soft/matlab/7.13";
string spikefilename = "X.mat";
string packedfilename = "glm_aic_results_";

app (file packlog, file packeddata) packdemo (string mcr, int nchn, file datafile, string dir)
{
  PackWrapper mcr nchn 60   @filename(datafile) dir;
}

app (file outdata,file result) causaltest (string mcr, int nchn, int target, int trigger, file datafile, file packed, string dir)
{
  causalCoreWrapper mcr nchn target trigger  @filename(datafile) @filename(packed) dir;
}

string WinDirs[] = ["trans11/Win2/","trans11/Win1/","trans11/Win3/","trans12/Win3/","trans12/Win2/","trans12/Win1/","trans11_s1/Win1/","trans11_s1/Win2/","trans11_s1/Win3/","trans11_s2/Win1/","trans11_s2/Win2/","trans11_s2/Win3/","trans11_s3/Win1/","trans11_s3/Win2/","trans11_s3/Win3/","trans11_s4/Win1/","trans11_s4/Win2/","trans11_s4/Win3/","trans11_s5/Win1/","trans11_s5/Win2/","trans11_s5/Win3/","trans11_s6/Win1/","trans11_s6/Win2/","trans11_s6/Win3/","trans11_s7/Win1/","trans11_s7/Win2/","trans11_s7/Win3/","trans11_s8/Win1/","trans11_s8/Win2/","trans11_s8/Win3/","trans11_s9/Win1/","trans11_s9/Win2/","trans11_s9/Win3/","trans11_s10/Win1/","trans11_s10/Win2/","trans11_s10/Win3/","trans11_s100/Win1/","trans11_s100/Win2/","trans11_s100/Win3/","trans11_s99/Win1/","trans11_s99/Win2/","trans11_s99/Win3/","trans11_s98/Win1/","trans11_s98/Win2/","trans11_s98/Win3/","trans11_s97/Win1/","trans11_s97/Win2/","trans11_s97/Win3/","trans11_s96/Win1/","trans11_s96/Win2/","trans11_s96/Win3/","trans11_s95/Win1/","trans11_s95/Win2/","trans11_s95/Win3/","trans11_s94/Win1/","trans11_s94/Win2/","trans11_s94/Win3/","trans11_s93/Win1/","trans11_s93/Win2/","trans11_s93/Win3/","trans11_s92/Win1/","trans11_s92/Win2/","trans11_s92/Win3/","trans11_s91/Win1/","trans11_s91/Win2/","trans11_s91/Win3/","trans11_s90/Win1/","trans11_s90/Win2/","trans11_s90/Win3/","trans11_s89/Win1/","trans11_s89/Win2/","trans11_s89/Win3/","trans11_s88/Win1/","trans11_s88/Win2/","trans11_s88/Win3/","trans11_s87/Win1/","trans11_s87/Win2/","trans11_s87/Win3/","trans11_s86/Win1/","trans11_s86/Win2/","trans11_s86/Win3/","trans11_s85/Win1/","trans11_s85/Win2/","trans11_s85/Win3/","trans11_s84/Win1/","trans11_s84/Win2/","trans11_s84/Win3/","trans11_s83/Win1/","trans11_s83/Win2/","trans11_s83/Win3/","trans11_s82/Win1/","trans11_s82/Win2/","trans11_s82/Win3/","trans11_s81/Win1/","trans11_s81/Win2/","trans11_s81/Win3/","trans11_s80/Win1/","trans11_s80/Win2/","trans11_s80/Win3/","trans11_s79/Win1/","trans11_s79/Win2/","trans11_s79/Win3/","trans11_s78/Win1/","trans11_s78/Win2/","trans11_s78/Win3/","trans11_s77/Win1/","trans11_s77/Win2/","trans11_s77/Win3/","trans11_s76/Win1/","trans11_s76/Win2/","trans11_s76/Win3/","trans11_s75/Win1/","trans11_s75/Win2/","trans11_s75/Win3/","trans11_s74/Win1/","trans11_s74/Win2/","trans11_s74/Win3/","trans11_s73/Win1/","trans11_s73/Win2/","trans11_s73/Win3/","trans11_s72/Win1/","trans11_s72/Win2/","trans11_s72/Win3/","trans11_s71/Win1/","trans11_s71/Win2/","trans11_s71/Win3/","trans11_s70/Win1/","trans11_s70/Win2/","trans11_s70/Win3/","trans11_s69/Win1/","trans11_s69/Win2/","trans11_s69/Win3/","trans11_s68/Win1/","trans11_s68/Win2/","trans11_s68/Win3/","trans11_s67/Win1/","trans11_s67/Win2/","trans11_s67/Win3/","trans11_s66/Win1/","trans11_s66/Win2/","trans11_s66/Win3/","trans11_s65/Win1/","trans11_s65/Win2/","trans11_s65/Win3/","trans11_s64/Win1/","trans11_s64/Win2/","trans11_s64/Win3/","trans11_s63/Win1/","trans11_s63/Win2/","trans11_s63/Win3/","trans11_s62/Win1/","trans11_s62/Win2/","trans11_s62/Win3/","trans11_s61/Win1/","trans11_s61/Win2/","trans11_s61/Win3/","trans11_s60/Win1/","trans11_s60/Win2/","trans11_s60/Win3/","trans11_s59/Win1/","trans11_s59/Win2/","trans11_s59/Win3/","trans11_s58/Win1/","trans11_s58/Win2/","trans11_s58/Win3/","trans11_s57/Win1/","trans11_s57/Win2/","trans11_s57/Win3/","trans11_s56/Win1/","trans11_s56/Win2/","trans11_s56/Win3/","trans11_s55/Win1/","trans11_s55/Win2/","trans11_s55/Win3/","trans11_s54/Win1/","trans11_s54/Win2/","trans11_s54/Win3/","trans11_s53/Win1/","trans11_s53/Win2/","trans11_s53/Win3/","trans11_s52/Win1/","trans11_s52/Win2/","trans11_s52/Win3/","trans11_s51/Win1/","trans11_s51/Win2/","trans11_s51/Win3/","trans11_s50/Win1/","trans11_s50/Win2/","trans11_s50/Win3/","trans11_s49/Win1/","trans11_s49/Win2/","trans11_s49/Win3/","trans11_s48/Win1/","trans11_s48/Win2/","trans11_s48/Win3/","trans11_s47/Win1/","trans11_s47/Win2/","trans11_s47/Win3/","trans11_s46/Win1/","trans11_s46/Win2/","trans11_s46/Win3/","trans11_s45/Win1/","trans11_s45/Win2/","trans11_s45/Win3/","trans11_s44/Win1/","trans11_s44/Win2/","trans11_s44/Win3/","trans11_s43/Win1/","trans11_s43/Win2/","trans11_s43/Win3/","trans11_s42/Win1/","trans11_s42/Win2/","trans11_s42/Win3/","trans11_s41/Win1/","trans11_s41/Win2/","trans11_s41/Win3/","trans11_s40/Win1/","trans11_s40/Win2/","trans11_s40/Win3/","trans11_s39/Win1/","trans11_s39/Win2/","trans11_s39/Win3/","trans11_s38/Win1/","trans11_s38/Win2/","trans11_s38/Win3/","trans11_s37/Win1/","trans11_s37/Win2/","trans11_s37/Win3/","trans11_s36/Win1/","trans11_s36/Win2/","trans11_s36/Win3/","trans11_s35/Win1/","trans11_s35/Win2/","trans11_s35/Win3/","trans11_s34/Win1/","trans11_s34/Win2/","trans11_s34/Win3/","trans11_s33/Win1/","trans11_s33/Win2/","trans11_s33/Win3/","trans11_s32/Win1/","trans11_s32/Win2/","trans11_s32/Win3/","trans11_s31/Win1/","trans11_s31/Win2/","trans11_s31/Win3/","trans11_s30/Win1/","trans11_s30/Win2/","trans11_s30/Win3/","trans11_s29/Win1/","trans11_s29/Win2/","trans11_s29/Win3/","trans11_s28/Win1/","trans11_s28/Win2/","trans11_s28/Win3/","trans11_s27/Win1/","trans11_s27/Win2/","trans11_s27/Win3/","trans11_s26/Win1/","trans11_s26/Win2/","trans11_s26/Win3/","trans11_s25/Win1/","trans11_s25/Win2/","trans11_s25/Win3/","trans11_s24/Win1/","trans11_s24/Win2/","trans11_s24/Win3/","trans11_s23/Win1/","trans11_s23/Win2/","trans11_s23/Win3/","trans11_s22/Win1/","trans11_s22/Win2/","trans11_s22/Win3/","trans11_s21/Win1/","trans11_s21/Win2/","trans11_s21/Win3/","trans11_s20/Win1/","trans11_s20/Win2/","trans11_s20/Win3/","trans11_s19/Win1/","trans11_s19/Win2/","trans11_s19/Win3/","trans11_s18/Win1/","trans11_s18/Win2/","trans11_s18/Win3/","trans11_s17/Win1/","trans11_s17/Win2/","trans11_s17/Win3/","trans11_s16/Win1/","trans11_s16/Win2/","trans11_s16/Win3/","trans11_s15/Win1/","trans11_s15/Win2/","trans11_s15/Win3/","trans11_s14/Win1/","trans11_s14/Win2/","trans11_s14/Win3/","trans11_s13/Win1/","trans11_s13/Win2/","trans11_s13/Win3/","trans11_s12/Win1/","trans11_s12/Win2/","trans11_s12/Win3/","trans11_s11/Win1/","trans11_s11/Win2/","trans11_s11/Win3/"];
string MaxChnWin[] = ["71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71","71"];

foreach win, iwin in WinDirs {
  # Map the input file and the packed file, both unique for each window
  file spikes <single_file_mapper; file=@strcat(win,spikefilename)>;
  file packedout <single_file_mapper; file=@strcat(win,packedfilename,MaxChnWin[iwin],".mat")>;
  file packedlog <single_file_mapper; file=@strcat(win,packedfilename,MaxChnWin[iwin],".log")>;

  (packedlog, packedout) = packdemo (MCRPath, , spikes, win);

  # determine the size of the trigger and chn (target) arrays
  int chnvec[] = [1:MaxChnWin[iwin]];
  int trgvec[] = [1:MaxChnWin[iwin]];

  foreach chn in chnvec {
       # Create a  Map the files for all triggers
       file causalout[] <simple_mapper; prefix=@strcat(win,"ECM_AIC_target", chn, "_trigger"),suffix=".mat")>;
       file causallog[] <simple_mapper; prefix=@strcat(win,"target", chn, "trigger"),suffix=".log")>;

       foreach trg in trgvec {

          (causallog[trg],causalout[trg]) = causaltest (MCRPath, , chn, trg, spikes, packedout, win);
       }
  }


}



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20121204/abde0811/attachment.html>

From lpesce at uchicago.edu  Wed Dec  5 09:18:39 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Wed, 5 Dec 2012 09:18:39 -0600
Subject: [Swift-user] Problems with PBS on Beagle for swift.0.94-2012.1102
Message-ID: <99E0C6E6-3FEE-42C8-8E9C-AC52ACDB1DBE@uchicago.edu>

From the logs:

Swift trunk swift-r6003 cog-r3497
RunID: 20121204-1504-llt6yhhb
Progress:  time: Tue, 04 Dec 2012 15:04:44 +0000
Progress:  time: Tue, 04 Dec 2012 15:04:53 +0000  Submitted:1  Active:1
Progress:  time: Tue, 04 Dec 2012 15:05:14 +0000  Active:2

<snip> 

Progress:  time: Wed, 05 Dec 2012 07:44:53 +0000  Active:1  Failed but can retry:1
Execution failed:
        Walltime exceeded
org.globus.cog.abstraction.impl.common.execution.JobException: Walltime exceeded (exit code: 513)
        at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:38)
        at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:91)
        at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:505)
        at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.stepNIO(AbstractStreamKarajanChannel.java:238)
        at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
        at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
 (exit code: 513)


By my reckoning it is 15 hours or 900 minutes or 54000 seconds

From my sites file:

<profile namespace="globus" key="maxTime">172800</profile>
 <profile namespace="globus" key="maxwalltime">16:40:00</profile>

Isn't maxwalltime the time expected from a single run app as opposed to the time allocated to the coaster? 

Has something changed in the way swift works? 

Lorenzo



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20121205/1d533e73/attachment.html>

From nbest at ci.uchicago.edu  Wed Dec  5 16:17:22 2012
From: nbest at ci.uchicago.edu (Neil Best)
Date: Wed, 5 Dec 2012 16:17:22 -0600
Subject: [Swift-user] Unix find as external mapper
Message-ID: <CANVU8H0i30V4nZchmf-FYrij2=98w0=CzFsgw12x26iudDsktw@mail.gmail.com>

I tried to use find as an external mapper in a way that seemed fairly
natural to me:

file monthly[] <ext; exec="find data/nc -mindepth 1 -type d -printf
'[%f] %p.nc\n'\|.txt'">;

When I run this find by itself at the command line the output looks like this:

$ find data/nc -mindepth 1 -type d -printf '[%f] %p.nc\n' | head
[197901] data/nc/197901.nc
[197903] data/nc/197903.nc
[197904] data/nc/197904.nc
[197902] data/nc/197902.nc
[197905] data/nc/197905.nc
[197906] data/nc/197906.nc
[197907] data/nc/197907.nc
[197908] data/nc/197908.nc
[197909] data/nc/197909.nc
[197910] data/nc/197910.nc

The find actually looks at directories.  These .nc files don't exist
yet, but each one will be an aggregation of the .nc files within the
folder of the same name using this command:

cdo mergetime data/nc/197901/*.single.nc data/nc/197901.nc

I thought I could then do a nested foreach over years and months to
execute the invocations.

I presume that the external mapper is treating the exec argument as a
command name and not parsing the arguments.  Using the other
parameters in the ext construct would force me to write a wrapper
script for the find invocation since all of the arguments must have
names apparently.

Do I have this right?  Maybe there is a more straightforward approach.


From wilde at mcs.anl.gov  Wed Dec  5 17:53:52 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 5 Dec 2012 17:53:52 -0600 (CST)
Subject: [Swift-user] Problems with PBS on Beagle for
 swift.0.94-2012.1102
In-Reply-To: <99E0C6E6-3FEE-42C8-8E9C-AC52ACDB1DBE@uchicago.edu>
Message-ID: <3361368.73157.1354751632562.JavaMail.root@zimbra.anl.gov>

Hi Lorenzo,

The "513" error comes from the coaster worker, and means that the app() call exceeded is maxwalltime.  This maxwalltime is specified first in the sites file (which applies to all apps on the site) and can be overridden for a given app by specifying maxwalltime on the app's entry in the tc file.

So, yes, if you specified maxwalltime as 16:40:00 in the sites file and did *not* override that in the tc file, then that is how long the app() call should be allowed to run on the compute node before the coaster worker kills it and returns a 513 error.

>From your logs, assuming the app started shortly after Swift started:

Started at: Tue, 04 Dec 2012 15:05:14
Got 513 at: Wed, 05 Dec 2012 07:44:53

24:00 - 15:05 = 8:55
              + 7:44
               15:99 = 16:39

which is what you asked for, mine the 1 min that Swift subtracts for overhead.

- Mike





----- Original Message -----
> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> To: swift-user at ci.uchicago.edu
> Cc: "Jason J. Pitt" <pittjj at uchicago.edu>
> Sent: Wednesday, December 5, 2012 9:18:39 AM
> Subject: [Swift-user] Problems with PBS on Beagle for swift.0.94-2012.1102
> From the logs:
> 
> 
> Swift trunk swift-r6003 cog-r3497
> RunID: 20121204-1504-llt6yhhb
> Progress: time: Tue, 04 Dec 2012 15:04:44 +0000
> Progress: time: Tue, 04 Dec 2012 15:04:53 +0000 Submitted:1 Active:1
> Progress: time: Tue, 04 Dec 2012 15:05:14 +0000 Active:2
> 
> 
> <snip>
> 
> 
> 
> Progress: time: Wed, 05 Dec 2012 07:44:53 +0000 Active:1 Failed but
> can retry:1
> Execution failed:
> Walltime exceeded
> org.globus.cog.abstraction.impl.common.execution.JobException:
> Walltime exceeded (exit code: 513)
> at
> org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:38)
> at
> org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:91)
> at
> org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:505)
> at
> org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.stepNIO(AbstractStreamKarajanChannel.java:238)
> at
> org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
> at
> org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
> (exit code: 513)
> 
> 
> 
> 
> By my reckoning it is 15 hours or 900 minutes or 54000 seconds
> 
> 
> From my sites file:
> 
> 
> 
> <profile namespace="globus" key="maxTime">172800</profile>
> 
> <profile namespace="globus" key="maxwalltime">16:40:00</profile>
> 
> 
> Isn't maxwalltime the time expected from a single run app as opposed
> to the time allocated to the coaster?
> 
> 
> Has something changed in the way swift works?
> 
> 
> Lorenzo
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Wed Dec  5 18:46:51 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 5 Dec 2012 18:46:51 -0600 (CST)
Subject: [Swift-user] Unix find as external mapper
In-Reply-To: <CANVU8H0i30V4nZchmf-FYrij2=98w0=CzFsgw12x26iudDsktw@mail.gmail.com>
Message-ID: <497698268.73215.1354754811015.JavaMail.root@zimbra.anl.gov>

Neil,

While your suggested syntax sounds reasonable, thats not how the ext mapper currently works.  You do indeed need to create a small wrapper script for such situations. The exec parameter is only used to specify the pathname of the mapper script (absolute or relative to the working dir in which you are running the swift command).

I'll file an enhancement ticket to record your suggestion, which in hindsight seems obvious, and which I like :)

Thanks,

- Mike

----- Original Message -----
> From: "Neil Best" <nbest at ci.uchicago.edu>
> To: swift-user at ci.uchicago.edu
> Sent: Wednesday, December 5, 2012 4:17:22 PM
> Subject: [Swift-user] Unix find as external mapper
> I tried to use find as an external mapper in a way that seemed fairly
> natural to me:
> 
> file monthly[] <ext; exec="find data/nc -mindepth 1 -type d -printf
> '[%f] %p.nc\n'\|.txt'">;
> 
> When I run this find by itself at the command line the output looks
> like this:
> 
> $ find data/nc -mindepth 1 -type d -printf '[%f] %p.nc\n' | head
> [197901] data/nc/197901.nc
> [197903] data/nc/197903.nc
> [197904] data/nc/197904.nc
> [197902] data/nc/197902.nc
> [197905] data/nc/197905.nc
> [197906] data/nc/197906.nc
> [197907] data/nc/197907.nc
> [197908] data/nc/197908.nc
> [197909] data/nc/197909.nc
> [197910] data/nc/197910.nc
> 
> The find actually looks at directories. These .nc files don't exist
> yet, but each one will be an aggregation of the .nc files within the
> folder of the same name using this command:
> 
> cdo mergetime data/nc/197901/*.single.nc data/nc/197901.nc
> 
> I thought I could then do a nested foreach over years and months to
> execute the invocations.
> 
> I presume that the external mapper is treating the exec argument as a
> command name and not parsing the arguments. Using the other
> parameters in the ext construct would force me to write a wrapper
> script for the find invocation since all of the arguments must have
> names apparently.
> 
> Do I have this right? Maybe there is a more straightforward approach.
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Wed Dec  5 18:55:38 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 5 Dec 2012 18:55:38 -0600 (CST)
Subject: [Swift-user] Unix find as external mapper
In-Reply-To: <497698268.73215.1354754811015.JavaMail.root@zimbra.anl.gov>
Message-ID: <361252650.73239.1354755338581.JavaMail.root@zimbra.anl.gov>

I need to qualify what I said in the prior posting.  Per the User Guide, an ext mapper needs to return two columns for the mapping, so for this format you almost always need a wrapper script to produce that format.

Neil, your suggestion would work for a mapper that mapped N elements of an array, with column 1 implicitly being [0], [1], ...  With that caveat, your suggestion makes sense for some other new but useful mapper type.

In Swift/T ( ) the approach to mappers is evolving somewhat differently, and we will discuss that with the community and provide some kind of backward compatibility option or conversion guide.

You might want to do this as follows, instead:

- declare "sh" as an app.

- pass your find command to the sh as the value of the -c argument, and have the app return stdout as its result.

- pass that to readData() into an array, and map the dataset with this array using the array_mapper:

string s[] = [ "a.txt", "b.txt", "c.txt" ];
file f[] <array_mapper;files=s>;

A simple example of the above belongs in the User Guide. I'll create a ticket for that as well.

- Mike


----- Original Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "Neil Best" <nbest at ci.uchicago.edu>
> Cc: swift-user at ci.uchicago.edu
> Sent: Wednesday, December 5, 2012 6:46:51 PM
> Subject: Re: [Swift-user] Unix find as external mapper
> Neil,
> 
> While your suggested syntax sounds reasonable, thats not how the ext
> mapper currently works. You do indeed need to create a small wrapper
> script for such situations. The exec parameter is only used to specify
> the pathname of the mapper script (absolute or relative to the working
> dir in which you are running the swift command).
> 
> I'll file an enhancement ticket to record your suggestion, which in
> hindsight seems obvious, and which I like :)
> 
> Thanks,
> 
> - Mike
> 
> ----- Original Message -----
> > From: "Neil Best" <nbest at ci.uchicago.edu>
> > To: swift-user at ci.uchicago.edu
> > Sent: Wednesday, December 5, 2012 4:17:22 PM
> > Subject: [Swift-user] Unix find as external mapper
> > I tried to use find as an external mapper in a way that seemed
> > fairly
> > natural to me:
> >
> > file monthly[] <ext; exec="find data/nc -mindepth 1 -type d -printf
> > '[%f] %p.nc\n'\|.txt'">;
> >
> > When I run this find by itself at the command line the output looks
> > like this:
> >
> > $ find data/nc -mindepth 1 -type d -printf '[%f] %p.nc\n' | head
> > [197901] data/nc/197901.nc
> > [197903] data/nc/197903.nc
> > [197904] data/nc/197904.nc
> > [197902] data/nc/197902.nc
> > [197905] data/nc/197905.nc
> > [197906] data/nc/197906.nc
> > [197907] data/nc/197907.nc
> > [197908] data/nc/197908.nc
> > [197909] data/nc/197909.nc
> > [197910] data/nc/197910.nc
> >
> > The find actually looks at directories. These .nc files don't exist
> > yet, but each one will be an aggregation of the .nc files within the
> > folder of the same name using this command:
> >
> > cdo mergetime data/nc/197901/*.single.nc data/nc/197901.nc
> >
> > I thought I could then do a nested foreach over years and months to
> > execute the invocations.
> >
> > I presume that the external mapper is treating the exec argument as
> > a
> > command name and not parsing the arguments. Using the other
> > parameters in the ext construct would force me to write a wrapper
> > script for the find invocation since all of the arguments must have
> > names apparently.
> >
> > Do I have this right? Maybe there is a more straightforward
> > approach.
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From lpesce at uchicago.edu  Wed Dec  5 19:40:32 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Wed, 5 Dec 2012 19:40:32 -0600
Subject: [Swift-user] Problems with PBS on Beagle for
	swift.0.94-2012.1102
In-Reply-To: <3361368.73157.1354751632562.JavaMail.root@zimbra.anl.gov>
References: <3361368.73157.1354751632562.JavaMail.root@zimbra.anl.gov>
Message-ID: <674196D0-35DA-4DF6-B9B6-BA81EE337BC3@uchicago.edu>


On Dec 5, 2012, at 5:53 PM, Michael Wilde wrote:

> Hi Lorenzo,
> 
> The "513" error comes from the coaster worker, and means that the app() call exceeded is maxwalltime.  This maxwalltime is specified first in the sites file (which applies to all apps on the site) and can be overridden for a given app by specifying maxwalltime on the app's entry in the tc file.
> 
> So, yes, if you specified maxwalltime as 16:40:00 in the sites file and did *not* override that in the tc file, then that is how long the app() call should be allowed to run on the compute node before the coaster worker kills it and returns a 513 error.

Yes, I understand that and the values match. 

However, I have been told that the first is the time the coasters use in the PBS submission and maxwalltime is used to plan how to run the apps, it is the time apps are expected to take inside of the coasters.

    <profile namespace="globus" key="maxTime">168000</profile>
    <profile namespace="globus" key="maxwalltime">2:0:00</profile>

This is how it seemed to work until version .94, has it changed? What do these two mean now?

Have I missed something?

> 
> From your logs, assuming the app started shortly after Swift started:
> 
> Started at: Tue, 04 Dec 2012 15:05:14
> Got 513 at: Wed, 05 Dec 2012 07:44:53
> 
> 24:00 - 15:05 = 8:55
>              + 7:44
>               15:99 = 16:39
> 
> which is what you asked for, mine the 1 min that Swift subtracts for overhead.
> 
> - Mike
> 
> 
> 
> 
> 
> ----- Original Message -----
>> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
>> To: swift-user at ci.uchicago.edu
>> Cc: "Jason J. Pitt" <pittjj at uchicago.edu>
>> Sent: Wednesday, December 5, 2012 9:18:39 AM
>> Subject: [Swift-user] Problems with PBS on Beagle for swift.0.94-2012.1102
>> From the logs:
>> 
>> 
>> Swift trunk swift-r6003 cog-r3497
>> RunID: 20121204-1504-llt6yhhb
>> Progress: time: Tue, 04 Dec 2012 15:04:44 +0000
>> Progress: time: Tue, 04 Dec 2012 15:04:53 +0000 Submitted:1 Active:1
>> Progress: time: Tue, 04 Dec 2012 15:05:14 +0000 Active:2
>> 
>> 
>> <snip>
>> 
>> 
>> 
>> Progress: time: Wed, 05 Dec 2012 07:44:53 +0000 Active:1 Failed but
>> can retry:1
>> Execution failed:
>> Walltime exceeded
>> org.globus.cog.abstraction.impl.common.execution.JobException:
>> Walltime exceeded (exit code: 513)
>> at
>> org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:38)
>> at
>> org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:91)
>> at
>> org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:505)
>> at
>> org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.stepNIO(AbstractStreamKarajanChannel.java:238)
>> at
>> org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
>> at
>> org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
>> (exit code: 513)
>> 
>> 
>> 
>> 
>> By my reckoning it is 15 hours or 900 minutes or 54000 seconds
>> 
>> 
>> From my sites file:
>> 
>> 
>> 
>> <profile namespace="globus" key="maxTime">172800</profile>
>> 
>> <profile namespace="globus" key="maxwalltime">16:40:00</profile>
>> 
>> 
>> Isn't maxwalltime the time expected from a single run app as opposed
>> to the time allocated to the coaster?
>> 
>> 
>> Has something changed in the way swift works?
>> 
>> 
>> Lorenzo
>> 
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 



From wilde at mcs.anl.gov  Wed Dec  5 20:00:55 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 5 Dec 2012 20:00:55 -0600 (CST)
Subject: [Swift-user] Problems with PBS on Beagle for
 swift.0.94-2012.1102
In-Reply-To: <674196D0-35DA-4DF6-B9B6-BA81EE337BC3@uchicago.edu>
Message-ID: <1444784359.73301.1354759255812.JavaMail.root@zimbra.anl.gov>


> Yes, I understand that and the values match.
> 
> However, I have been told that the first is the time the coasters use
> in the PBS submission and maxwalltime is used to plan how to run the
> apps, it is the time apps are expected to take inside of the coasters.
> 
> <profile namespace="globus" key="maxTime">168000</profile>
> <profile namespace="globus" key="maxwalltime">2:0:00</profile>
> 
> This is how it seemed to work until version .94, has it changed? What
> do these two mean now?

No, nothing has changed in that regard to my knowledge for 0.94. What you say above is correct, and is the same as what I said in my reply, right?

But the maxwalltime you said you used in your sites file was 16:40:00, not 2:0:00, right?

> Have I missed something?

Im confused by your question, Lorenzo.  You said in the original email on this thread that your site file specified:

  <profile namespace="globus" key="maxTime">172800</profile>
  <profile namespace="globus" key="maxwalltime">16:40:00</profile>

and it seemed like you were asking why your app() invocation was terminated by a 513 error after what you said was "By my reckoning it is 15 hours or 900 minutes or 54000 seconds"

But as far as I can tell, your app() was terminated exactly when I would expect it to be terminated, after a walltime of 16:39:00. This is the way Swift and Coasters worked in 0.93 and has not to my knowledge changed in 0.94. Nothing I see in the info you provided looks to me like either an error or a change in behavior.

Can you clarify your question?

- Mike




From wilde at mcs.anl.gov  Wed Dec  5 20:55:36 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 5 Dec 2012 20:55:36 -0600 (CST)
Subject: [Swift-user] Problems with PBS on Beagle for
 swift.0.94-2012.1102
In-Reply-To: <1444784359.73301.1354759255812.JavaMail.root@zimbra.anl.gov>
Message-ID: <1191342697.73355.1354762536521.JavaMail.root@zimbra.anl.gov>

To close this issue: I discussed the problem with Lorenzo off-list and realized the confusion was that in 0.93, maxwalltime was just used by the coaster provider to fit app() invocations into coaster worker jobs. At some point after 0.93 though, coasters started enforcing maxwalltime. I think this resolves the question. Lorenzo said off-list

> Actually, I think that now I understand it. I always assumed that it
> was an "indicative" quantity and not an actual lethal threshold.
> It kind of makes sense both way, but it works as lethal too.
> Maybe I never hit the kill zone before so I never realized it.

Ah - I see that part of the confusion. Yes, the 513 error and the terminating of coaster jobs that go over their maxwalltime *is* new since 0.93.  Its been in trunk for many months I think (since Spring?) but if youve been running 0.93 till now then yes, the "kill" part is a change.

The problem with the old "advisory" semantics is that when PBS killed the coaster worker, it was much harder for Swift to recover cleanly in all cases. So this 513-kill was instituted to make things both more consistent and more reliable.

Sorry for missing that part of the change. Ive been running trunk typically, so I forgot that the 513-kill was new since 0.93.

- Mike


From hockyg at gmail.com  Wed Dec  5 21:23:12 2012
From: hockyg at gmail.com (Glen Hocky)
Date: Wed, 5 Dec 2012 22:23:12 -0500
Subject: [Swift-user] Problems with PBS on Beagle for
	swift.0.94-2012.1102
In-Reply-To: <1191342697.73355.1354762536521.JavaMail.root@zimbra.anl.gov>
References: <1444784359.73301.1354759255812.JavaMail.root@zimbra.anl.gov>
	<1191342697.73355.1354762536521.JavaMail.root@zimbra.anl.gov>
Message-ID: <CAGMfakmZo+qWHnrv_V2H7ZOUE8T+G8C9ULnxvuCG-ZhPnH3BQg@mail.gmail.com>

I also noticed this behavior recently, so I'm glad this was explained. I
have a follow up question

You already discussed the maxwalltime parameter as set by the sites file
and as set by the tc file.

At some point, I added the following to my app() call

    profile "maxwalltime"=maxwalltime;

where maxwalltime is an argument passed to app

This was supposed to allow individual app calls to have separate
(expected) durations.

Is coasters currently taking this into account or only the values set in
sites and tc?

Thanks
Glen





On Wed, Dec 5, 2012 at 9:55 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> To close this issue: I discussed the problem with Lorenzo off-list and
> realized the confusion was that in 0.93, maxwalltime was just used by the
> coaster provider to fit app() invocations into coaster worker jobs. At some
> point after 0.93 though, coasters started enforcing maxwalltime. I think
> this resolves the question. Lorenzo said off-list
>
> > Actually, I think that now I understand it. I always assumed that it
> > was an "indicative" quantity and not an actual lethal threshold.
> > It kind of makes sense both way, but it works as lethal too.
> > Maybe I never hit the kill zone before so I never realized it.
>
> Ah - I see that part of the confusion. Yes, the 513 error and the
> terminating of coaster jobs that go over their maxwalltime *is* new since
> 0.93.  Its been in trunk for many months I think (since Spring?) but if
> youve been running 0.93 till now then yes, the "kill" part is a change.
>
> The problem with the old "advisory" semantics is that when PBS killed the
> coaster worker, it was much harder for Swift to recover cleanly in all
> cases. So this 513-kill was instituted to make things both more consistent
> and more reliable.
>
> Sorry for missing that part of the change. Ive been running trunk
> typically, so I forgot that the 513-kill was new since 0.93.
>
> - Mike
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20121205/4f392259/attachment.html>

From wilde at mcs.anl.gov  Wed Dec  5 21:43:11 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 5 Dec 2012 21:43:11 -0600 (CST)
Subject: [Swift-user] Problems with PBS on Beagle for
 swift.0.94-2012.1102
In-Reply-To: <CAGMfakmZo+qWHnrv_V2H7ZOUE8T+G8C9ULnxvuCG-ZhPnH3BQg@mail.gmail.com>
Message-ID: <1420897224.73381.1354765391728.JavaMail.root@zimbra.anl.gov>

Glen,

Swift *should* be honoring profile parameters passed on the app() declaration as if they were passed on the tc or sites entries.  Profile entries passed on the app() call should override those passed on the tc, which override the site entries.

If any of that is not working, I would consider it a bug.

- Mike


----- Original Message -----
> From: "Glen Hocky" <hockyg at gmail.com>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Lorenzo Pesce" <lpesce at uchicago.edu>, "Jason J. Pitt" <pittjj at uchicago.edu>, "Swift User Discussion List"
> <swift-user at ci.uchicago.edu>
> Sent: Wednesday, December 5, 2012 9:23:12 PM
> Subject: Re: [Swift-user] Problems with PBS on Beagle for swift.0.94-2012.1102
> I also noticed this behavior recently, so I'm glad this was explained.
> I have a follow up question
> 
> 
> You already discussed the maxwalltime parameter as set by the sites
> file and as set by the tc file.
> 
> 
> 
> At some point, I added the following to my app() call
> 
> 
> 
> profile "maxwalltime"=maxwalltime;
> 
> 
> where maxwalltime is an argument passed to app
> 
> 
> 
> This was supposed to allow individual app calls to have separate
> (expected) durations.
> 
> 
> Is coasters currently taking this into account or only the values set
> in sites and tc?
> 
> 
> Thanks
> Glen
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Wed, Dec 5, 2012 at 9:55 PM, Michael Wilde < wilde at mcs.anl.gov >
> wrote:
> 
> 
> To close this issue: I discussed the problem with Lorenzo off-list and
> realized the confusion was that in 0.93, maxwalltime was just used by
> the coaster provider to fit app() invocations into coaster worker
> jobs. At some point after 0.93 though, coasters started enforcing
> maxwalltime. I think this resolves the question. Lorenzo said off-list
> 
> > Actually, I think that now I understand it. I always assumed that it
> > was an "indicative" quantity and not an actual lethal threshold.
> > It kind of makes sense both way, but it works as lethal too.
> > Maybe I never hit the kill zone before so I never realized it.
> 
> Ah - I see that part of the confusion. Yes, the 513 error and the
> terminating of coaster jobs that go over their maxwalltime *is* new
> since 0.93. Its been in trunk for many months I think (since Spring?)
> but if youve been running 0.93 till now then yes, the "kill" part is a
> change.
> 
> The problem with the old "advisory" semantics is that when PBS killed
> the coaster worker, it was much harder for Swift to recover cleanly in
> all cases. So this 513-kill was instituted to make things both more
> consistent and more reliable.
> 
> Sorry for missing that part of the change. Ive been running trunk
> typically, so I forgot that the 513-kill was new since 0.93.
> 
> 
> 
> - Mike
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From lpesce at uchicago.edu  Thu Dec  6 10:39:57 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Thu, 6 Dec 2012 10:39:57 -0600
Subject: [Swift-user] Memory is a serious issue
Message-ID: <891F999F-C125-41CD-A0FE-CEF8E0F7D968@uchicago.edu>

Hi! 

It is me again. 

Discounting for my manifest inability of using swift intelligently, it seems that to runs simulations we need *a lot* of RAM (8+ GB for the large sims, say > 10^6 app runs, which is taxing on the login nodes). Any idea about how we can reduce out memory footprint?

Otherwise, I just sent a 300K sim as a small test and it seems to work. (I call it a small test because the next one is going to be in the 10G size).

Lorenzo



From hategan at mcs.anl.gov  Thu Dec  6 16:52:05 2012
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 06 Dec 2012 14:52:05 -0800
Subject: [Swift-user] Memory is a serious issue
In-Reply-To: <891F999F-C125-41CD-A0FE-CEF8E0F7D968@uchicago.edu>
References: <891F999F-C125-41CD-A0FE-CEF8E0F7D968@uchicago.edu>
Message-ID: <1354834325.28279.3.camel@blabla>

Unfortunately, there are no aces down the sleeve at this point.

When it's ready, Swift/T might provide better scalability.
If I have time, I might upgrade the Swift/K engine to the same end.

The simple things you can do now are:
- Don't run swift on login nodes. This won't solve the memory problem,
but it won't tax the login nodes.
- De-couple the coaster service (if you are using that) from swift by
running it non-local. This might give you some more memory for swift
alone.

Mihael

On Thu, 2012-12-06 at 10:39 -0600, Lorenzo Pesce wrote:
> Hi! 
> 
> It is me again. 
> 
> Discounting for my manifest inability of using swift intelligently, it seems that to runs simulations we need *a lot* of RAM (8+ GB for the large sims, say > 10^6 app runs, which is taxing on the login nodes). Any idea about how we can reduce out memory footprint?
> 
> Otherwise, I just sent a 300K sim as a small test and it seems to work. (I call it a small test because the next one is going to be in the 10G size).
> 
> Lorenzo
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user




From lpesce at uchicago.edu  Thu Dec  6 19:35:57 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Thu, 6 Dec 2012 19:35:57 -0600
Subject: [Swift-user] apps running with different number of cores or in
	different numbers per node
Message-ID: <63168A99-2809-403C-A395-8A5B621C0EEB@uchicago.edu>

This is the trick

I have to run a complex workflow and different parts of the workflow have require different amount of memory and or threads.
Thus when I pack them, the number per node should vary.

But the number of app per nodes is defined in the sites file, which is one per run.

How do I do it? Can I do it?

Lorenzo



From wilde at mcs.anl.gov  Thu Dec  6 20:47:39 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 6 Dec 2012 20:47:39 -0600
Subject: [Swift-user] apps running with different number of cores or in
 different numbers per node
In-Reply-To: <63168A99-2809-403C-A395-8A5B621C0EEB@uchicago.edu>
References: <63168A99-2809-403C-A395-8A5B621C0EEB@uchicago.edu>
Message-ID: <CAF1H9xqx1UD7L1Lkj0UNei9CNe23vettZfKXEWS2_NHeLeTMxw@mail.gmail.com>

To do this, define multiple sites, one for each value of jobsPerNode
etc. Each site should have a unique uri on the sites pool entry
exection provider tag.

For an example of two pools on the same cluster see bin/runswift  in
the latest RCC tutorial in
https://svn.ci.uchicago.edu/svn/SwiftTutorials.



On 12/6/12, Lorenzo Pesce <lpesce at uchicago.edu> wrote:
> This is the trick
>
> I have to run a complex workflow and different parts of the workflow have
> require different amount of memory and or threads.
> Thus when I pack them, the number per node should vary.
>
> But the number of app per nodes is defined in the sites file, which is one
> per run.
>
> How do I do it? Can I do it?
>
> Lorenzo
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>

-- 
Sent from my mobile device


From nbest at ci.uchicago.edu  Fri Dec  7 10:51:43 2012
From: nbest at ci.uchicago.edu (Neil Best)
Date: Fri, 7 Dec 2012 10:51:43 -0600
Subject: [Swift-user] many-to-one mapping
Message-ID: <CANVU8H3uXwUDH+Rta9Z4MGJX=cT7KgqWNoH3PvHOWzsBeyu_ww@mail.gmail.com>

Is it possible to declare a mapping within a foreach loop so that
different realizations of that mapping are driven by the loop
parameter?

I tried this:

foreach year in [1979:2011] {
  ncList[ year] = @filenames(
    file foo[]<structured_regexp_mapper;
      source= nc1,
      match= @strcat( "^data/nc/", year, "../narr-a_221_", year,
"...._..00_000.single.nc$");>)
  annual[ year]= cdo( "-O mergetime", ncList[ year]);
  tmin[ year]= cdo( "-O daymin -selname,TMP", @annual[ year]);
  tmax[ year]= cdo( "-O daymax -selname,TMP", @annual[ year]);
  precip[ year]= cdo( "-O daysum -selname,APCP", @annual[ year]);
  solar[ year]= cdo( "-O daymean -selname,DSWRF", @annual[ year]);
}

The idea is that the annual[] files are the product of merging ~3000
3-hour reports.  The set of reports for each year is a subset of all
of the 3-hour reports across all of the years.

I have a feeling that my syntax is not legal where I am trying to work
in terms of first class objects and functional programming, but I am
not sure to what extent Swift can accept these constructs.  Is it
clear what I am trying to do?  Is there an idiom for this?  I don't
see any exmaples where a mapper is parameterized by a variable within
a loop.

Complete listing is here:
https://github.com/RDCEP/narr2wth/blob/master/narr.swift


From nbest at ci.uchicago.edu  Fri Dec  7 11:21:35 2012
From: nbest at ci.uchicago.edu (Neil Best)
Date: Fri, 7 Dec 2012 11:21:35 -0600
Subject: [Swift-user] many-to-one mapping
In-Reply-To: <CANVU8H3uXwUDH+Rta9Z4MGJX=cT7KgqWNoH3PvHOWzsBeyu_ww@mail.gmail.com>
References: <CANVU8H3uXwUDH+Rta9Z4MGJX=cT7KgqWNoH3PvHOWzsBeyu_ww@mail.gmail.com>
Message-ID: <CANVU8H0u3PXORtFi32bxJJ=wSRxbe2m2qwapLg2nk-LsBpr1QA@mail.gmail.com>

On Fri, Dec 7, 2012 at 10:51 AM, Neil Best <nbest at ci.uchicago.edu> wrote:
> Is it possible to declare a mapping within a foreach loop so that
> different realizations of that mapping are driven by the loop
> parameter?
>
> I tried this:
>
> foreach year in [1979:2011] {
>   ncList[ year] = @filenames(
>     file foo[]<structured_regexp_mapper;
>       source= nc1,
>       match= @strcat( "^data/nc/", year, "../narr-a_221_", year,
> "...._..00_000.single.nc$");>)
>   annual[ year]= cdo( "-O mergetime", ncList[ year]);
>   tmin[ year]= cdo( "-O daymin -selname,TMP", @annual[ year]);
>   tmax[ year]= cdo( "-O daymax -selname,TMP", @annual[ year]);
>   precip[ year]= cdo( "-O daysum -selname,APCP", @annual[ year]);
>   solar[ year]= cdo( "-O daymean -selname,DSWRF", @annual[ year]);
> }
>
> The idea is that the annual[] files are the product of merging ~3000
> 3-hour reports.  The set of reports for each year is a subset of all
> of the 3-hour reports across all of the years.
>
> I have a feeling that my syntax is not legal where I am trying to work
> in terms of first class objects and functional programming, but I am
> not sure to what extent Swift can accept these constructs.  Is it
> clear what I am trying to do?  Is there an idiom for this?  I don't
> see any exmaples where a mapper is parameterized by a variable within
> a loop.
>
> Complete listing is here:
> https://github.com/RDCEP/narr2wth/blob/master/narr.swift

I meant to mention that Swift says the following when I try to run the above:

Could not compile SwiftScript source: line 88:10: expecting ')', found 'foo'


From lpesce at uchicago.edu  Fri Dec  7 14:28:38 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Fri, 7 Dec 2012 14:28:38 -0600
Subject: [Swift-user] again about absolute and relative paths
Message-ID: <AE0FCE10-FBA7-4EDC-B981-0B5D99F4EF6C@uchicago.edu>

Hi --

I have another problem. 

The calculations I working on right now (different from the one I was working on before or before) has this problem:

input files are in say /lustre/beagle/megagenomics/input_files

while my work directory is

/lustre/beagle/lpesce/test

It is easy to connect the input files as strings and pass them down to the apps. No problem.

Now I have a different problem, I want to pass them down as files so I can use them both in direct and staging, depending upon the system.

For all the intermediary and output files, I use a relative path, relative to /lustre/beagle/lpesce/test
for all apps I use absolute paths and that works fine too

e.g. :

export BAMdir="/lustre/beagle/megagenomics"

and from the swift script

string rootFileName [] = ${swiftSampleIDs};

foreach rootf in rootFileName {
       #string inBAM=@strcat("${BAMdir}/",rootf,".bam");
       # Absolute path
       file TCGABAM <single_file_mapper; file=@strcat(${BAMdir},"/",rootf,".bam")>;
       # relative path
       file outBAM <single_file_mapper; file=@strcat(rootf,"/",rootf,".bam")>;
       file outBAMgrp <single_file_mapper; file=@strcat(rootf,"/",rootf,".bam.grp")>;
       file LogGATK <single_file_mapper; file=@strcat(rootf,"/",rootf,".log")>;
       (LogGATK,outBAM,outBAMgrp) = GATKpost (inBAM,rootf);



However, when I use absolute paths for input files, it seems to come out garbled (like the first / is cut, so that it can't find the file). 
Am I missing something in the logic of this?






From hategan at mcs.anl.gov  Fri Dec  7 14:35:00 2012
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 07 Dec 2012 12:35:00 -0800
Subject: [Swift-user] many-to-one mapping
In-Reply-To: <CANVU8H3uXwUDH+Rta9Z4MGJX=cT7KgqWNoH3PvHOWzsBeyu_ww@mail.gmail.com>
References: <CANVU8H3uXwUDH+Rta9Z4MGJX=cT7KgqWNoH3PvHOWzsBeyu_ww@mail.gmail.com>
Message-ID: <1354912500.21068.1.camel@blabla>

Yes, but you can't invoke stuff inside mapping specs. So you need
something like:

foreach {
   string m = @strcat(...);
   file[] foo <...;..., match = m,...>;
}

Mihael

On Fri, 2012-12-07 at 10:51 -0600, Neil Best wrote:
> Is it possible to declare a mapping within a foreach loop so that
> different realizations of that mapping are driven by the loop
> parameter?
> 
> I tried this:
> 
> foreach year in [1979:2011] {
>   ncList[ year] = @filenames(
>     file foo[]<structured_regexp_mapper;
>       source= nc1,
>       match= @strcat( "^data/nc/", year, "../narr-a_221_", year,
> "...._..00_000.single.nc$");>)
>   annual[ year]= cdo( "-O mergetime", ncList[ year]);
>   tmin[ year]= cdo( "-O daymin -selname,TMP", @annual[ year]);
>   tmax[ year]= cdo( "-O daymax -selname,TMP", @annual[ year]);
>   precip[ year]= cdo( "-O daysum -selname,APCP", @annual[ year]);
>   solar[ year]= cdo( "-O daymean -selname,DSWRF", @annual[ year]);
> }
> 
> The idea is that the annual[] files are the product of merging ~3000
> 3-hour reports.  The set of reports for each year is a subset of all
> of the 3-hour reports across all of the years.
> 
> I have a feeling that my syntax is not legal where I am trying to work
> in terms of first class objects and functional programming, but I am
> not sure to what extent Swift can accept these constructs.  Is it
> clear what I am trying to do?  Is there an idiom for this?  I don't
> see any exmaples where a mapper is parameterized by a variable within
> a loop.
> 
> Complete listing is here:
> https://github.com/RDCEP/narr2wth/blob/master/narr.swift
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user




From hategan at mcs.anl.gov  Fri Dec  7 15:08:48 2012
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 07 Dec 2012 13:08:48 -0800
Subject: [Swift-user] again about absolute and relative paths
In-Reply-To: <AE0FCE10-FBA7-4EDC-B981-0B5D99F4EF6C@uchicago.edu>
References: <AE0FCE10-FBA7-4EDC-B981-0B5D99F4EF6C@uchicago.edu>
Message-ID: <1354914528.22056.5.camel@blabla>

On Fri, 2012-12-07 at 14:28 -0600, Lorenzo Pesce wrote:

> 
> However, when I use absolute paths for input files, it seems to come out garbled (like the first / is cut, so that it can't find the file). 
> Am I missing something in the logic of this?

Yes. You should not make assumptions about the location of files on the
remote side based on the way they are on the swift side. Pass that
location to your executable using @filename() instead.

Now the internal details. Swift restricts files on the app execution
side to the work directory. In order to do that for absolute files and
prevent clashes (to some extent), it does something equivalent to
chroot(workdir), effectively removing the first "/" from the absolute
path. The file is staged in to the new location before the app is run.

Mihael



From nbest at ci.uchicago.edu  Fri Dec  7 15:08:50 2012
From: nbest at ci.uchicago.edu (Neil Best)
Date: Fri, 7 Dec 2012 15:08:50 -0600
Subject: [Swift-user] many-to-one mapping
In-Reply-To: <1354912500.21068.1.camel@blabla>
References: <CANVU8H3uXwUDH+Rta9Z4MGJX=cT7KgqWNoH3PvHOWzsBeyu_ww@mail.gmail.com>
	<1354912500.21068.1.camel@blabla>
Message-ID: <CANVU8H2FFZCgofUMSL+K9pQdA5UFQohtyZRFu=Nt8BVPtPiUqg@mail.gmail.com>

On Fri, Dec 7, 2012 at 2:35 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>
> foreach {
>    string m = @strcat(...);
>    file[] foo <...;..., match = m,...>;
> }
>

Shouldn't m also be an array?

Also, I don't recognize "file[] foo" in your notation.

I tried this:

foreach year in [1979:2011] {
  m[ year] = @strcat( "^data/nc/", year, "../narr-a_221_", year,
"...._..00_000.single.nc$");
  ncList[ year] = @filenames(
    file foo[]<structured_regexp_mapper;
      source= nc1,
      match= m[ year];>);
  annual[ year]= cdo( "-O mergetime", ncList[ year]);
}

It still doesn't like that the mapper is an argument to @filenames(), I guess.

Could not compile SwiftScript source: line 90:10: expecting ')', found 'foo'

I hope it is clear what I am trying to do.  Each member of ncList[]
would be a long string enumerating the input files for the cdo command
and must be different for each year.


From hategan at mcs.anl.gov  Fri Dec  7 15:21:33 2012
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 07 Dec 2012 13:21:33 -0800
Subject: [Swift-user] many-to-one mapping
In-Reply-To: <CANVU8H2FFZCgofUMSL+K9pQdA5UFQohtyZRFu=Nt8BVPtPiUqg@mail.gmail.com>
References: <CANVU8H3uXwUDH+Rta9Z4MGJX=cT7KgqWNoH3PvHOWzsBeyu_ww@mail.gmail.com>
	<1354912500.21068.1.camel@blabla>
	<CANVU8H2FFZCgofUMSL+K9pQdA5UFQohtyZRFu=Nt8BVPtPiUqg@mail.gmail.com>
Message-ID: <1354915293.22056.14.camel@blabla>

On Fri, 2012-12-07 at 15:08 -0600, Neil Best wrote:
> On Fri, Dec 7, 2012 at 2:35 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >
> > foreach {
> >    string m = @strcat(...);
> >    file[] foo <...;..., match = m,...>;
> > }
> >
> 
> Shouldn't m also be an array?

It's the pattern, so I'm guessing no.

> 
> Also, I don't recognize "file[] foo" in your notation.

Oh, I see. I'll add that you cannot declare variables inside invocation
argument lists. And I'm becoming a bit unsure about my previous
statement. You might be able to invoke stuff in mapper params. But let's
try what's more likely to work:

foreach year in [1979:2011] {
  string m = @strcat( "^data/nc/",year, "../narr-a_221_", year,
"...._..00_000.single.nc$");
  file[] foo <structured_regexp_mapper; source= nc1, match= m>;
  ncList[year] = @filenames(foo);
  ...
}

or you could do ncList[year] = foo; and then apply @filenames in the app
call.

Mihael

> 
> I tried this:
> 
> foreach year in [1979:2011] {
>   m[ year] = @strcat( "^data/nc/", year, "../narr-a_221_", year,
> "...._..00_000.single.nc$");
>   ncList[ year] = @filenames(
>     file foo[]<structured_regexp_mapper;
>       source= nc1,
>       match= m[ year];>);
>   annual[ year]= cdo( "-O mergetime", ncList[ year]);
> }
> 
> It still doesn't like that the mapper is an argument to @filenames(), I guess.
> 
> Could not compile SwiftScript source: line 90:10: expecting ')', found 'foo'
> 
> I hope it is clear what I am trying to do.  Each member of ncList[]
> would be a long string enumerating the input files for the cdo command
> and must be different for each year.




From lpesce at uchicago.edu  Fri Dec  7 16:00:16 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Fri, 7 Dec 2012 16:00:16 -0600
Subject: [Swift-user] again about absolute and relative paths
In-Reply-To: <1354914528.22056.5.camel@blabla>
References: <AE0FCE10-FBA7-4EDC-B981-0B5D99F4EF6C@uchicago.edu>
	<1354914528.22056.5.camel@blabla>
Message-ID: <542895A0-AD87-4F80-B2B3-B9258BE64F8B@uchicago.edu>

Hi Mihael, 

Thanks a lot for your quick reply. 

> Yes. You should not make assumptions about the location of files on the
> remote side based on the way they are on the swift side. Pass that
> location to your executable using @filename() instead.

thanks and yes I am doing that.

> Now the internal details. Swift restricts files on the app execution
> side to the work directory. In order to do that for absolute files and
> prevent clashes (to some extent), it does something equivalent to
> chroot(workdir), effectively removing the first "/" from the absolute
> path. The file is staged in to the new location before the app is run.

So, how will it find the files if I use direct? would not the filename have the "/"
chopped off and therefore point to the wrong location?

(which is the last error message I have seen before I gave up and moved to using 
strings and not files, but that approach prevents me from using any form of staging
without some complex machinery)

I could conceivably use a relative path (not sure whether it works since it will go upwards, but I can try).
Can I define a path relative to SWIFT_USERHOME  or USER_HOME in swift?


> 
> Mihael
> 



From hategan at mcs.anl.gov  Fri Dec  7 16:18:28 2012
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 07 Dec 2012 14:18:28 -0800
Subject: [Swift-user] again about absolute and relative paths
In-Reply-To: <542895A0-AD87-4F80-B2B3-B9258BE64F8B@uchicago.edu>
References: <AE0FCE10-FBA7-4EDC-B981-0B5D99F4EF6C@uchicago.edu>
	<1354914528.22056.5.camel@blabla>
	<542895A0-AD87-4F80-B2B3-B9258BE64F8B@uchicago.edu>
Message-ID: <1354918708.25239.4.camel@blabla>

I'm slightly confused.

You are saying that using @filename gives you an error. Can you be more
explicit about what that error is?

Without CDM, if you have file f<"/a/b/c/name">, your app will find that
file in "a/b/c/name". In other words, there will be a path "a/b/c/name"
in your work directory, so if your app only did "ls -al a/b/c/name" it
should succeed.

If you use CDM direct mode, it should know enough to put back the "/".

Mihael

On Fri, 2012-12-07 at 16:00 -0600, Lorenzo Pesce wrote:
> Hi Mihael, 
> 
> Thanks a lot for your quick reply. 
> 
> > Yes. You should not make assumptions about the location of files on the
> > remote side based on the way they are on the swift side. Pass that
> > location to your executable using @filename() instead.
> 
> thanks and yes I am doing that.
> 
> > Now the internal details. Swift restricts files on the app execution
> > side to the work directory. In order to do that for absolute files and
> > prevent clashes (to some extent), it does something equivalent to
> > chroot(workdir), effectively removing the first "/" from the absolute
> > path. The file is staged in to the new location before the app is run.
> 
> So, how will it find the files if I use direct? would not the filename have the "/"
> chopped off and therefore point to the wrong location?
> 
> (which is the last error message I have seen before I gave up and moved to using 
> strings and not files, but that approach prevents me from using any form of staging
> without some complex machinery)
> 
> I could conceivably use a relative path (not sure whether it works since it will go upwards, but I can try).
> Can I define a path relative to SWIFT_USERHOME  or USER_HOME in swift?
> 
> 
> > 
> > Mihael
> > 
> 




From wozniak at mcs.anl.gov  Sat Dec  8 11:16:08 2012
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Sat, 08 Dec 2012 10:16:08 -0700
Subject: [Swift-user] again about absolute and relative paths
In-Reply-To: <1354918708.25239.4.camel@blabla>
References: <AE0FCE10-FBA7-4EDC-B981-0B5D99F4EF6C@uchicago.edu>
	<1354914528.22056.5.camel@blabla>
	<542895A0-AD87-4F80-B2B3-B9258BE64F8B@uchicago.edu>
	<1354918708.25239.4.camel@blabla>
Message-ID: <50C375D8.50902@mcs.anl.gov>

On 12/07/2012 03:18 PM, Mihael Hategan wrote:
> I'm slightly confused.
>
> You are saying that using @filename gives you an error. Can you be more
> explicit about what that error is?

> If you use CDM direct mode, it should know enough to put back the "/".

Lorenzo, the -info files should contain explicit information about what 
CDM does and what paths it operates on.  If you want me to take a look, 
let me know where to find the logs.

-- 
Justin M Wozniak



From marialemos72 at gmail.com  Mon Dec 10 03:53:30 2012
From: marialemos72 at gmail.com (WorldCIST - CFP)
Date: Mon, 10 Dec 2012 09:53:30 +0000
Subject: [Swift-user] CISTI'2013 Doctoral Symposium - CFP, Lisbon,
	June 19 - 23, 2013
Message-ID: <20121210095240.478B67CC0B2@mailrelay.anl.gov>


***************************************************************************************************
CISTI'2013 DOCTORAL SYMPOSIUM
8th Iberian Conference on Information Systems and Technologies
Lisbn, Portugal, June 19 - 23, 2013
http://www.aisti.eu/cisti2013/index.php?option=com_content&view=article&id=64&Itemid=68&lang=en
***************************************************************************************************

INTRODUCTION

The purpose of CISTI'2013?s Doctoral Symposium (http://www.aisti.eu/cisti2013/index.php?option=com_content&view=article&id=64&Itemid=68&lang=en) is to provide graduate students a setting where they can, informally, expose and discuss their work, collecting valuable expert opinions and sharing new ideas, methods and applications. The Doctoral Symposium is an excellent opportunity for PhD students to present and discuss their work in a Workshop format. Each presentation will be evaluated by a panel composed by at least three Information Systems and Technologies experts. 

CONTRIBUTIONS SUBMISSION

The Doctoral Symposium is opened to PhD students whose research area includes the themes proposed for this Conference. Submissions must include an extended abstract (maximum 4 pages), following the Conference style guide. All selected contributions will be handed out along with the Conference Proceedings, in CD with an ISBN. These contributions will be send for indexation by EBSCO, and EI-Compendex.

Submissions must include the field, the PhD institution and the number of months devoted to the development of the work. Additionally, they should include in a clear and succinct manner:

    ?    The problem approached and its significance or relevance
    ?    The research objectives and related investigation topics
    ?    A brief display of what is already known
    ?    A proposed solution methodology for the problem
    ?    Expected results

IMPORTANT DATES

?    Data limite para submiss?o de propostas: 15 de Fevereiro de 2013
?    Notifica??o de aceita??o: 29 de Mar?o de 2013
?    Data limite para apresenta??o das vers?es finais: 12 de Abril de 2013
?    Pagamento da inscri??o, para garantir a inclus?o da contribui??o aceite nas actas da confer?ncia: 12 de Abril de 2013

SCIENTIFIC AND ORGANIZING COMMITTEE

Manuel P?rez Cota, Universidad de Vigo (Chair)
Adolfo Lozano Tello, Universidad de Extremadura
Alberto J. Bugar?n Diz, Universidad de Santiago de Compostela
?lvaro Rocha, Universidade Fernando Pessoa
Ana Maria Ramalho Correia, Universidade Nova de Lisboa, ISEGI
Ant?nio Palma dos Reis, Universidade T?cnica de Lisboa, ISEG
Arturo Mendez Pen?n, Universidade de Vigo
Carlos Ferr?s Sexto, Universidad de Santiago de Compostela
David Fonseca, Universidad Ram?n Llul
Ernesto Redondo, Universidad Polit?cnica de Catalu?a
Feliz Gouveia, Universidade Fernando Pessoa
Francisco Restivo, Universidade Cat?lica Portuguesa - Braga
Guilhermina Miranda, Universidade de Lisboa
Gonzalo Cuevas Agust?n, Universidad Polit?cnica de Madrid
H?ctor Jorge Garc?a Neder, Universidad Tecnol?gica NacioNal
Jo?o ?lvaro Carvalho, Universidade do Minho
Jo?o Barroso, Universidade de Tr?s-os-Montes e Alto Douro
J?rg Thomaschewski, University of Applied Sciences of Emden-Leer
Jos? Antonio Calvo-Manzano Villal?n, Universidad Polit?cnica de Madrid
Jos? Bulas Cruz, Universidade de Tr?s-os-Montes e Alto Douro
Jos? Tribolet, Universidade T?cnica de Lisboa, IST
Leandro Rodr?guez Li?ares, Universidade de Vigo
Lu?s Paulo Reis, Universidade do Minho
Mar?a Jos? Lado Touri?o, Universidade de Vigo
Maria Manuela Cruz Cunha, Instituto Polit?cnico do C?vado e do Ave
Marco Painho, Universidade Nova de Lisboa, ISEGI
Mario Alberto Groppo, Universidad Tecnol?gica Nacional
Nuno Ribeiro, Universidade Fernando Pessoa
Pilar Mareca, Universidade Polit?cnica de Madrid
Ramiro Gon?alves, Universidade de Tr?s-os-Montes e Alto Douro
Tomas San Feliu Gilabert, Universidad Polit?cnica de Madrid
Vicente Alcober, Universidad Polit?cnica de Madrid

-
CISTI'2013 Team
http://www.aisti.eu/cisti2013
aistic at gmail.com

-------------------



From marialemos72 at gmail.com  Mon Dec 10 03:57:47 2012
From: marialemos72 at gmail.com (Maria Lemos)
Date: Mon, 10 Dec 2012 09:57:47 +0000
Subject: [Swift-user] CISTI'2013 Doctoral Symposium - CFP, Lisbon,
	June 19 - 23, 2013
Message-ID: <20121210095656.5CCC07CC0B0@mailrelay.anl.gov>


***************************************************************************************************
CISTI'2013 DOCTORAL SYMPOSIUM
8th Iberian Conference on Information Systems and Technologies
Lisbn, Portugal, June 19 - 23, 2013
http://www.aisti.eu/cisti2013/index.php?option=com_content&view=article&id=64&Itemid=68&lang=en
***************************************************************************************************

INTRODUCTION

The purpose of CISTI'2013?s Doctoral Symposium (http://www.aisti.eu/cisti2013/index.php?option=com_content&view=article&id=64&Itemid=68&lang=en) is to provide graduate students a setting where they can, informally, expose and discuss their work, collecting valuable expert opinions and sharing new ideas, methods and applications. The Doctoral Symposium is an excellent opportunity for PhD students to present and discuss their work in a Workshop format. Each presentation will be evaluated by a panel composed by at least three Information Systems and Technologies experts. 

CONTRIBUTIONS SUBMISSION

The Doctoral Symposium is opened to PhD students whose research area includes the themes proposed for this Conference. Submissions must include an extended abstract (maximum 4 pages), following the Conference style guide. All selected contributions will be handed out along with the Conference Proceedings, in CD with an ISBN. These contributions will be send for indexation by EBSCO, and EI-Compendex.

Submissions must include the field, the PhD institution and the number of months devoted to the development of the work. Additionally, they should include in a clear and succinct manner:

    ?    The problem approached and its significance or relevance
    ?    The research objectives and related investigation topics
    ?    A brief display of what is already known
    ?    A proposed solution methodology for the problem
    ?    Expected results

IMPORTANT DATES

?    Data limite para submiss?o de propostas: 15 de Fevereiro de 2013
?    Notifica??o de aceita??o: 29 de Mar?o de 2013
?    Data limite para apresenta??o das vers?es finais: 12 de Abril de 2013
?    Pagamento da inscri??o, para garantir a inclus?o da contribui??o aceite nas actas da confer?ncia: 12 de Abril de 2013

SCIENTIFIC AND ORGANIZING COMMITTEE

Manuel P?rez Cota, Universidad de Vigo (Chair)
Adolfo Lozano Tello, Universidad de Extremadura
Alberto J. Bugar?n Diz, Universidad de Santiago de Compostela
?lvaro Rocha, Universidade Fernando Pessoa
Ana Maria Ramalho Correia, Universidade Nova de Lisboa, ISEGI
Ant?nio Palma dos Reis, Universidade T?cnica de Lisboa, ISEG
Arturo Mendez Pen?n, Universidade de Vigo
Carlos Ferr?s Sexto, Universidad de Santiago de Compostela
David Fonseca, Universidad Ram?n Llul
Ernesto Redondo, Universidad Polit?cnica de Catalu?a
Feliz Gouveia, Universidade Fernando Pessoa
Francisco Restivo, Universidade Cat?lica Portuguesa - Braga
Guilhermina Miranda, Universidade de Lisboa
Gonzalo Cuevas Agust?n, Universidad Polit?cnica de Madrid
H?ctor Jorge Garc?a Neder, Universidad Tecnol?gica NacioNal
Jo?o ?lvaro Carvalho, Universidade do Minho
Jo?o Barroso, Universidade de Tr?s-os-Montes e Alto Douro
J?rg Thomaschewski, University of Applied Sciences of Emden-Leer
Jos? Antonio Calvo-Manzano Villal?n, Universidad Polit?cnica de Madrid
Jos? Bulas Cruz, Universidade de Tr?s-os-Montes e Alto Douro
Jos? Tribolet, Universidade T?cnica de Lisboa, IST
Leandro Rodr?guez Li?ares, Universidade de Vigo
Lu?s Paulo Reis, Universidade do Minho
Mar?a Jos? Lado Touri?o, Universidade de Vigo
Maria Manuela Cruz Cunha, Instituto Polit?cnico do C?vado e do Ave
Marco Painho, Universidade Nova de Lisboa, ISEGI
Mario Alberto Groppo, Universidad Tecnol?gica Nacional
Nuno Ribeiro, Universidade Fernando Pessoa
Pilar Mareca, Universidade Polit?cnica de Madrid
Ramiro Gon?alves, Universidade de Tr?s-os-Montes e Alto Douro
Tomas San Feliu Gilabert, Universidad Polit?cnica de Madrid
Vicente Alcober, Universidad Polit?cnica de Madrid

-
CISTI'2013 Team
http://www.aisti.eu/cisti2013
aistic at gmail.com

-------------------



From nbest at ci.uchicago.edu  Mon Dec 10 13:58:24 2012
From: nbest at ci.uchicago.edu (Neil Best)
Date: Mon, 10 Dec 2012 13:58:24 -0600
Subject: [Swift-user] many-to-one mapping
In-Reply-To: <1354915293.22056.14.camel@blabla>
References: <CANVU8H3uXwUDH+Rta9Z4MGJX=cT7KgqWNoH3PvHOWzsBeyu_ww@mail.gmail.com>
	<1354912500.21068.1.camel@blabla>
	<CANVU8H2FFZCgofUMSL+K9pQdA5UFQohtyZRFu=Nt8BVPtPiUqg@mail.gmail.com>
	<1354915293.22056.14.camel@blabla>
Message-ID: <CANVU8H38_H3gSa5AOnRMoPxnyL74ptSQP8mkspPXX9e_Pu4LqA@mail.gmail.com>

On Fri, Dec 7, 2012 at 3:21 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> On Fri, 2012-12-07 at 15:08 -0600, Neil Best wrote:
>> On Fri, Dec 7, 2012 at 2:35 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>> >
>> > foreach {
>> >    string m = @strcat(...);
>> >    file[] foo <...;..., match = m,...>;
>> > }
>> >
>>
>> Shouldn't m also be an array?
>
> It's the pattern, so I'm guessing no.

I think there is a disconnect here because in my scenario there would
have to be a different regex for each year.  I need to match the
annual subset of all inputs (which span many years) to the
corresponding year.

Similarly the members of foo[] would have to be different for each
year.  I called it foo originally because I was treating it like a
throw-away.  Essentially what I need is a list of lists that maps
years (the loop variable) to subsets of files from nc1[].  I'm not
sure whether Swift needs an actual mapping of these files or if
passing the long string that is the concatenation of all of their
names to the leaf app is adequate.

So now I am doing this:

app (file n1) cdo( string op, string ifiles) {
  cdo op ifiles @n1;
}

file annual[]<simple_mapper;
  location="data/nc/annual/swift",
  suffix=".nc">;

string m[];

foreach year in [1979:2011] {
  m[ year] = @strcat( "^(data/nc/", year, "../narr-a_221_", year,
"...._..00_000.single.nc)$");
  file foo[]<structured_regexp_mapper;
    source= nc1,
    match= m[ year],
    transform= "\\1">;
  annual[ year]= cdo( "-O mergetime", @filename( foo));
}

where nc1 is a mapped array of files from a previously successful
Swift run, so the processing statements are commented out.

I suspect that the files in foo[] need to be mapped, not just have
their names passed as a concatenated string to the app, which brings
me back to the opening question: how to do a many to one mapping,
especially in the context of a looping parameter.  I hope I am asking
the right question, but I would not be surprised if something salient
about Swift semantics is eluding me.  Thanks.


From hategan at mcs.anl.gov  Mon Dec 10 20:23:15 2012
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 10 Dec 2012 18:23:15 -0800
Subject: [Swift-user] many-to-one mapping
In-Reply-To: <CANVU8H38_H3gSa5AOnRMoPxnyL74ptSQP8mkspPXX9e_Pu4LqA@mail.gmail.com>
References: <CANVU8H3uXwUDH+Rta9Z4MGJX=cT7KgqWNoH3PvHOWzsBeyu_ww@mail.gmail.com>
	<1354912500.21068.1.camel@blabla>
	<CANVU8H2FFZCgofUMSL+K9pQdA5UFQohtyZRFu=Nt8BVPtPiUqg@mail.gmail.com>
	<1354915293.22056.14.camel@blabla>
	<CANVU8H38_H3gSa5AOnRMoPxnyL74ptSQP8mkspPXX9e_Pu4LqA@mail.gmail.com>
Message-ID: <1355192595.22430.19.camel@blabla>

Inline. All the way to the end.

On Mon, 2012-12-10 at 13:58 -0600, Neil Best wrote:
> On Fri, Dec 7, 2012 at 3:21 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> > On Fri, 2012-12-07 at 15:08 -0600, Neil Best wrote:
> >> On Fri, Dec 7, 2012 at 2:35 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >> >
> >> > foreach {
> >> >    string m = @strcat(...);
> >> >    file[] foo <...;..., match = m,...>;
> >> > }
> >> >
> >>
> >> Shouldn't m also be an array?
> >
> > It's the pattern, so I'm guessing no.
> 
> I think there is a disconnect here because in my scenario there would
> have to be a different regex for each year.  I need to match the
> annual subset of all inputs (which span many years) to the
> corresponding year.

Right. Declaring m inside the foreach year {} says "give me a different
m for each year".

> 
> Similarly the members of foo[] would have to be different for each
> year.  I called it foo originally because I was treating it like a
> throw-away.  Essentially what I need is a list of lists that maps
> years (the loop variable) to subsets of files from nc1[].  I'm not
> sure whether Swift needs an actual mapping of these files or if
> passing the long string that is the concatenation of all of their
> names to the leaf app is adequate.
> 
> So now I am doing this:
> 
> app (file n1) cdo( string op, string ifiles) {
>   cdo op ifiles @n1;
> }
> 
> file annual[]<simple_mapper;
>   location="data/nc/annual/swift",
>   suffix=".nc">;
> 
> string m[];

You don't really need the array outside of the foreach here because of
what I mentioned above. 

> 
> foreach year in [1979:2011] {
>   m[ year] = @strcat( "^(data/nc/", year, "../narr-a_221_", year,
> "...._..00_000.single.nc)$");
>   file foo[]<structured_regexp_mapper;
>     source= nc1,
>     match= m[ year],
>     transform= "\\1">;
>   annual[ year]= cdo( "-O mergetime", @filename( foo));
> }
> 
> where nc1 is a mapped array of files from a previously successful
> Swift run, so the processing statements are commented out.

> I suspect that the files in foo[] need to be mapped, not just have
> their names passed as a concatenated string to the app, which brings
> me back to the opening question: how to do a many to one mapping,
> especially in the context of a looping parameter.  I hope I am asking
> the right question, but I would not be surprised if something salient
> about Swift semantics is eluding me.  Thanks.

Ok, so I think that there's some unclarity about what the structured
regexp mapper does. It doesn't select a subset of an existing collection
of data. It only generates a new set of file names from EACH of the file
names of the source array. The "match" parameter seems to be a misnomer.
It is meant to indicate the parts of the source file name that should be
selected, not select which files should be considered and which
shouldn't. If ANY of the file names in src fail to match against the
provided regexp, swift throws an error.

What you probably want is the filesys mapper:

file foo[] <filesys_mapper; location=@strcat("data/nc/", year, "/"),
pattern=@strcat("narr-a_221_1_", year, "????_??00_000.single.nc")>;

If you have that inside the foreach year loop, and if year = 2000, it
should give you all files that would be returned by "ls
data/nc/2000/narr-a_221_1_2000????_??_00_000.single.nc". Am I
understanding what you need correctly this time?

Mihael



From nbest at ci.uchicago.edu  Tue Dec 11 14:02:45 2012
From: nbest at ci.uchicago.edu (Neil Best)
Date: Tue, 11 Dec 2012 14:02:45 -0600
Subject: [Swift-user] many-to-one mapping
In-Reply-To: <1355192595.22430.19.camel@blabla>
References: <CANVU8H3uXwUDH+Rta9Z4MGJX=cT7KgqWNoH3PvHOWzsBeyu_ww@mail.gmail.com>
	<1354912500.21068.1.camel@blabla>
	<CANVU8H2FFZCgofUMSL+K9pQdA5UFQohtyZRFu=Nt8BVPtPiUqg@mail.gmail.com>
	<1354915293.22056.14.camel@blabla>
	<CANVU8H38_H3gSa5AOnRMoPxnyL74ptSQP8mkspPXX9e_Pu4LqA@mail.gmail.com>
	<1355192595.22430.19.camel@blabla>
Message-ID: <CANVU8H3YXNTZAgA=_J3fQ67DTSBhnvcerZisG26FQRrpTD1mGA@mail.gmail.com>

On Mon, Dec 10, 2012 at 8:23 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> file foo[] <filesys_mapper; location=@strcat("data/nc/", year, "/"),
> pattern=@strcat("narr-a_221_1_", year, "????_??00_000.single.nc")>;

This is starting to make some sense to me.  Unfortunately the data for
a given year spans 12 monthly directories (data/nc/197901,
data/nc/197902, etc.).  Therefore I would like to be able to say:

foreach year in [1979:2011] {
  file thisYear[]<filesys_mapper;
    location= @strcat( "data/nc/", year, "*"),
    pattern= @strcat( "narr-a_221_", year, "????_??00_000.single.nc")>;
. . .

("*" glob at the end of location) but this does not work, nor do I
expect it to.  I am sure that location is meant to be a literal, not a
glob.  That's why I thought I needed a regex.  Is there a work-around
other than the obvious rearrangement of the data?


From hategan at mcs.anl.gov  Tue Dec 11 20:11:43 2012
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 11 Dec 2012 18:11:43 -0800
Subject: [Swift-user] many-to-one mapping
In-Reply-To: <CANVU8H3YXNTZAgA=_J3fQ67DTSBhnvcerZisG26FQRrpTD1mGA@mail.gmail.com>
References: <CANVU8H3uXwUDH+Rta9Z4MGJX=cT7KgqWNoH3PvHOWzsBeyu_ww@mail.gmail.com>
	<1354912500.21068.1.camel@blabla>
	<CANVU8H2FFZCgofUMSL+K9pQdA5UFQohtyZRFu=Nt8BVPtPiUqg@mail.gmail.com>
	<1354915293.22056.14.camel@blabla>
	<CANVU8H38_H3gSa5AOnRMoPxnyL74ptSQP8mkspPXX9e_Pu4LqA@mail.gmail.com>
	<1355192595.22430.19.camel@blabla>
	<CANVU8H3YXNTZAgA=_J3fQ67DTSBhnvcerZisG26FQRrpTD1mGA@mail.gmail.com>
Message-ID: <1355278303.3146.3.camel@blabla>

On Tue, 2012-12-11 at 14:02 -0600, Neil Best wrote:
> On Mon, Dec 10, 2012 at 8:23 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> > file foo[] <filesys_mapper; location=@strcat("data/nc/", year, "/"),
> > pattern=@strcat("narr-a_221_1_", year, "????_??00_000.single.nc")>;
> 
> This is starting to make some sense to me.  Unfortunately the data for
> a given year spans 12 monthly directories (data/nc/197901,
> data/nc/197902, etc.).  Therefore I would like to be able to say:
> 
> foreach year in [1979:2011] {
>   file thisYear[]<filesys_mapper;
>     location= @strcat( "data/nc/", year, "*"),
>     pattern= @strcat( "narr-a_221_", year, "????_??00_000.single.nc")>;
> . . .
> 
> ("*" glob at the end of location) but this does not work, nor do I
> expect it to.  I am sure that location is meant to be a literal, not a
> glob.  That's why I thought I needed a regex.  Is there a work-around
> other than the obvious rearrangement of the data?

Not that I can think of.

I filed this for now:
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=897



From lpesce at uchicago.edu  Thu Dec 13 11:14:36 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Thu, 13 Dec 2012 11:14:36 -0600
Subject: [Swift-user] Silly question about mappers
Message-ID: <1104858D-CC32-47CA-B754-A9977CED7AED@uchicago.edu>

When I use a simple mapper, integers are mapped as 4 digit numbers, e.g., as "0003".
Is there a way to force the mapper to just use "3", i.e., discard the preceding "0"?

Thanks

Lorenzo

From wilde at mcs.anl.gov  Thu Dec 13 11:33:54 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 13 Dec 2012 11:33:54 -0600 (CST)
Subject: [Swift-user] Silly question about mappers
In-Reply-To: <1104858D-CC32-47CA-B754-A9977CED7AED@uchicago.edu>
Message-ID: <2021485233.90885.1355420034069.JavaMail.root@zimbra.anl.gov>

There is a simple_mapper option "padding=" the enables you to set the number of digits to be used in the generated names. This needs to be documented in the User Guide.

- Mike

----- Original Message -----
> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> To: swift-user at ci.uchicago.edu
> Sent: Thursday, December 13, 2012 11:14:36 AM
> Subject: [Swift-user] Silly question about mappers
> When I use a simple mapper, integers are mapped as 4 digit numbers,
> e.g., as "0003".
> Is there a way to force the mapper to just use "3", i.e., discard the
> preceding "0"?
> 
> Thanks
> 
> Lorenzo
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From lpesce at uchicago.edu  Thu Dec 13 12:05:20 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Thu, 13 Dec 2012 12:05:20 -0600
Subject: [Swift-user] CoasterService fatal error (Urgent if possible)
Message-ID: <0171E7E8-DE73-42F2-85EB-68E0A93AECC6@uchicago.edu>

What does this mean?

CoasterService fatal error:
Exception caught in block processor
java.util.NoSuchElementException
        at org.globus.cog.abstraction.coaster.service.job.manager.SortedJobSet$1.next(SortedJobSet.java:177)
        at org.globus.cog.abstraction.coaster.service.job.manager.SortedJobSet$1.next(SortedJobSet.java:157)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.allocateBlocks(BlockQueueProcessor.java:600)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:725)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:133)
Exception caught in block processor
java.util.NoSuchElementException
        at org.globus.cog.abstraction.coaster.service.job.manager.SortedJobSet$1.next(SortedJobSet.java:177)
        at org.globus.cog.abstraction.coaster.service.job.manager.SortedJobSet$1.next(SortedJobSet.java:157)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.allocateBlocks(BlockQueueProcessor.java:600)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:725)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:133)



From wilde at mcs.anl.gov  Thu Dec 13 12:10:19 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 13 Dec 2012 12:10:19 -0600 (CST)
Subject: [Swift-user] CoasterService fatal error (Urgent if possible)
In-Reply-To: <0171E7E8-DE73-42F2-85EB-68E0A93AECC6@uchicago.edu>
Message-ID: <518754309.91078.1355422219259.JavaMail.root@zimbra.anl.gov>

This looks like a Swift bug.

Can you send us (or point is to) the run's .log file, as a ticket to swift-support?

- Mike


----- Original Message -----
> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> To: swift-user at ci.uchicago.edu
> Sent: Thursday, December 13, 2012 12:05:20 PM
> Subject: [Swift-user] CoasterService fatal error (Urgent if possible)
> What does this mean?
> 
> CoasterService fatal error:
> Exception caught in block processor
> java.util.NoSuchElementException
> at
> org.globus.cog.abstraction.coaster.service.job.manager.SortedJobSet$1.next(SortedJobSet.java:177)
> at
> org.globus.cog.abstraction.coaster.service.job.manager.SortedJobSet$1.next(SortedJobSet.java:157)
> at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.allocateBlocks(BlockQueueProcessor.java:600)
> at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:725)
> at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:133)
> Exception caught in block processor
> java.util.NoSuchElementException
> at
> org.globus.cog.abstraction.coaster.service.job.manager.SortedJobSet$1.next(SortedJobSet.java:177)
> at
> org.globus.cog.abstraction.coaster.service.job.manager.SortedJobSet$1.next(SortedJobSet.java:157)
> at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.allocateBlocks(BlockQueueProcessor.java:600)
> at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:725)
> at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:133)
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From lpesce at uchicago.edu  Mon Dec 17 15:05:39 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Mon, 17 Dec 2012 15:05:39 -0600
Subject: [Swift-user] Mystery of staging out glut
Message-ID: <FB887BB3-BABE-4C65-B855-CD1394429111@uchicago.edu>

I am not sure I understand this.

I am running a job which involved a few million tasks over about 10K cores. It seems that when the swift process reaches some memory size (about 10 GB) it starts to produce a lot of stage out glut (right now I have about 5K tasks in stage out), then the number of active tasks drops and eventually I have to kill the job.

Have anyone seen a similar behavior?


This is what the screen trace looks like:

Progress:  time: Mon, 17 Dec 2012 20:56:26 +0000  Selecting site:27662  Stage in:4641  Submitting:1  Active:5430  Stage out:4044  Finished in previous run:1253574  Finished successfully:29963
Progress:  time: Mon, 17 Dec 2012 20:56:27 +0000  Selecting site:27611  Stage in:4664  Submitting:2  Active:5385  Stage out:4065  Finished in previous run:1253574  Finished successfully:30014
Progress:  time: Mon, 17 Dec 2012 20:56:28 +0000  Selecting site:27576  Stage in:4678  Submitting:1  Active:5406  Stage out:4032  Finished in previous run:1253574  Finished successfully:30048
Progress:  time: Mon, 17 Dec 2012 20:56:52 +0000  Selecting site:27574  Stage in:4678  Submitted:1  Active:5408  Stage out:4029  Finished in previous run:1253574  Finished successfully:30051
Progress:  time: Mon, 17 Dec 2012 20:56:53 +0000  Selecting site:27543  Stage in:4688  Submitting:2  Active:5361  Stage out:4066  Finished in previous run:1253574  Finished successfully:30081
Progress:  time: Mon, 17 Dec 2012 20:57:18 +0000  Selecting site:27522  Stage in:4702  Submitted:2  Active:5367  Stage out:4046  Finished in previous run:1253574  Finished successfully:30102
Progress:  time: Mon, 17 Dec 2012 20:57:19 +0000  Selecting site:27457  Stage in:4725  Submitted:1  Active:5349  Stage out:4041  Finished in previous run:1253574  Finished successfully:30168
Progress:  time: Mon, 17 Dec 2012 20:57:43 +0000  Selecting site:27439  Stage in:4733  Submitting:1  Active:5358  Stage out:4025  Finished in previous run:1253574  Finished successfully:30185
Progress:  time: Mon, 17 Dec 2012 20:57:44 +0000  Selecting site:27360  Stage in:4786  Submitting:2  Active:5295  Stage out:4034  Finished in previous run:1253574  Finished successfully:30264
Progress:  time: Mon, 17 Dec 2012 20:58:11 +0000  Selecting site:27351  Stage in:4788  Submitting:2  Active:5302  Stage out:4021  Finished in previous run:1253574  Finished successfully:30277
Progress:  time: Mon, 17 Dec 2012 20:58:12 +0000  Selecting site:27291  Stage in:4834  Active:5228  Stage out:4054  Finished in previous run:1253574  Finished successfully:30334
Progress:  time: Mon, 17 Dec 2012 20:58:37 +0000  Selecting site:27193  Stage in:4895  Active:5263  Checking status:1  Stage out:3958  Finished in previous run:1253574  Finished successfully:30431
Progress:  time: Mon, 17 Dec 2012 20:58:38 +0000  Selecting site:27188  Stage in:4895  Submitting:2  Submitted:1  Active:5183  Stage out:4035  Finished in previous run:1253574  Finished successfully:30437
Progress:  time: Mon, 17 Dec 2012 20:58:39 +0000  Selecting site:27184  Stage in:4898  Submitted:3  Active:5184  Stage out:4031  Finished in previous run:1253574  Finished successfully:30441
Progress:  time: Mon, 17 Dec 2012 20:58:41 +0000  Selecting site:27182  Stage in:4897  Submitting:2  Submitted:1  Active:5187  Stage out:4030  Finished in previous run:1253574  Finished successfully:30442
Progress:  time: Mon, 17 Dec 2012 20:58:43 +0000  Selecting site:27182  Stage in:4897  Submitting:2  Active:5188  Stage out:4030  Finished in previous run:1253574  Finished successfully:30442
Progress:  time: Mon, 17 Dec 2012 20:58:51 +0000  Selecting site:27182  Stage in:4897  Submitting:1  Submitted:1  Active:5188  Stage out:4030  Finished in previous run:1253574  Finished successfully:30442
Progress:  time: Mon, 17 Dec 2012 20:58:52 +0000  Selecting site:27182  Stage in:4897  Submitted:2  Active:5188  Stage out:4029  Finished in previous run:1253574  Finished successfully:30443
Progress:  time: Mon, 17 Dec 2012 20:58:53 +0000  Selecting site:27181  Stage in:4897  Submitting:1  Active:5190  Stage out:4029  Finished in previous run:1253574  Finished successfully:30443
Progress:  time: Mon, 17 Dec 2012 20:59:01 +0000  Selecting site:27179  Stage in:4898  Submitting:2  Active:5190  Stage out:4027  Finished in previous run:1253574  Finished successfully:30445
Progress:  time: Mon, 17 Dec 2012 20:59:26 +0000  Selecting site:27179  Stage in:4898  Submitting:2  Active:5190  Stage out:4027  Finished in previous run:1253574  Finished successfully:30445
Progress:  time: Mon, 17 Dec 2012 20:59:27 +0000  Selecting site:27102  Stage in:4952  Submitted:1  Active:5051  Stage out:4113  Finished in previous run:1253574  Finished successfully:30522
Progress:  time: Mon, 17 Dec 2012 20:59:52 +0000  Selecting site:27092  Stage in:4955  Submitting:3  Active:5055  Checking status:1  Stage out:4103  Finished in previous run:1253574  Finished successfully:30532
Progress:  time: Mon, 17 Dec 2012 20:59:56 +0000  Selecting site:27092  Stage in:4955  Submitting:3  Active:4989  Stage out:4170  Finished in previous run:1253574  Finished successfully:30532
Progress:  time: Mon, 17 Dec 2012 20:59:58 +0000  Selecting site:27092  Stage in:4954  Submitting:1  Submitted:2  Active:4990  Stage out:4170  Finished in previous run:1253574  Finished successfully:30532
Progress:  time: Mon, 17 Dec 2012 20:59:59 +0000  Selecting site:27091  Stage in:4955  Submitted:1  Active:4991  Checking status:1  Stage out:4169  Finished in previous run:1253574  Finished successfully:30533
Progress:  time: Mon, 17 Dec 2012 21:00:01 +0000  Selecting site:27091  Stage in:4955  Active:4991  Stage out:4171  Finished in previous run:1253574  Finished successfully:30533
Progress:  time: Mon, 17 Dec 2012 21:00:02 +0000  Selecting site:27091  Stage in:4955  Active:4990  Checking status:1  Stage out:4171  Finished in previous run:1253574  Finished successfully:30533
Progress:  time: Mon, 17 Dec 2012 21:00:07 +0000  Selecting site:27091  Stage in:4955  Active:4990  Stage out:4171  Finished in previous run:1253574  Finished successfully:30534
Progress:  time: Mon, 17 Dec 2012 21:00:09 +0000  Selecting site:27089  Stage in:4957  Active:4990  Stage out:4169  Finished in previous run:1253574  Finished successfully:30536
Progress:  time: Mon, 17 Dec 2012 21:00:10 +0000  Selecting site:27087  Stage in:4958  Submitting:1  Active:4990  Stage out:4167  Finished in previous run:1253574  Finished successfully:30538
Progress:  time: Mon, 17 Dec 2012 21:00:11 +0000  Selecting site:27085  Stage in:4960  Submitted:1  Active:4990  Stage out:4166  Finished in previous run:1253574  Finished successfully:30539
Progress:  time: Mon, 17 Dec 2012 21:00:37 +0000  Selecting site:27065  Stage in:4967  Submitting:1  Active:4949  Stage out:4198  Finished in previous run:1253574  Finished successfully:30561
Progress:  time: Mon, 17 Dec 2012 21:00:38 +0000  Selecting site:27018  Stage in:5000  Submitting:1  Active:4810  Stage out:4305  Finished in previous run:1253574  Finished successfully:30607
Progress:  time: Mon, 17 Dec 2012 21:01:03 +0000  Selecting site:26988  Stage in:5003  Submitting:6  Active:4828  Stage out:4279  Finished in previous run:1253574  Finished successfully:30637
Progress:  time: Mon, 17 Dec 2012 21:01:04 +0000  Selecting site:26963  Stage in:5016  Active:4726  Checking status:1  Stage out:4374  Finished in previous run:1253574  Finished successfully:30661
Progress:  time: Mon, 17 Dec 2012 21:01:29 +0000  Selecting site:26930  Stage in:5037  Submitted:4  Active:4724  Stage out:4352  Finished in previous run:1253574  Finished successfully:30694
Progress:  time: Mon, 17 Dec 2012 21:01:30 +0000  Selecting site:26864  Stage in:5083  Active:4649  Stage out:4385  Finished in previous run:1253574  Finished successfully:30760
Progress:  time: Mon, 17 Dec 2012 21:01:55 +0000  Selecting site:26858  Stage in:5085  Submitting:1  Active:4651  Stage out:4380  Finished in previous run:1253574  Finished successfully:30766
Progress:  time: Mon, 17 Dec 2012 21:01:56 +0000  Selecting site:26811  Stage in:5119  Submitted:1  Active:4561  Stage out:4436  Finished in previous run:1253574  Finished successfully:30813
Progress:  time: Mon, 17 Dec 2012 21:02:21 +0000  Selecting site:26805  Stage in:5124  Active:4562  Stage out:4431  Finished in previous run:1253574  Finished successfully:30819
Progress:  time: Mon, 17 Dec 2012 21:02:22 +0000  Selecting site:26745  Stage in:5157  Active:4487  Stage out:4473  Finished in previous run:1253574  Finished successfully:30879
Progress:  time: Mon, 17 Dec 2012 21:02:47 +0000  Selecting site:26704  Stage in:5165  Submitted:3  Active:4514  Stage out:4435  Finished in previous run:1253574  Finished successfully:30920
Progress:  time: Mon, 17 Dec 2012 21:02:48 +0000  Selecting site:26671  Stage in:5171  Submitted:1  Active:4450  Stage out:4495  Finished in previous run:1253574  Finished successfully:30953
Progress:  time: Mon, 17 Dec 2012 21:03:13 +0000  Selecting site:26667  Stage in:5172  Submitting:1  Active:4451  Stage out:4494  Finished in previous run:1253574  Finished successfully:30956
Progress:  time: Mon, 17 Dec 2012 21:03:38 +0000  Selecting site:26603  Stage in:5197  Submitting:2  Active:4380  Stage out:4539  Finished in previous run:1253574  Finished successfully:31020
Progress:  time: Mon, 17 Dec 2012 21:03:39 +0000  Selecting site:26562  Stage in:5220  Submitted:2  Active:4305  Stage out:4591  Finished in previous run:1253574  Finished successfully:31061
Progress:  time: Mon, 17 Dec 2012 21:04:04 +0000  Selecting site:26559  Stage in:5222  Submitted:1  Active:4307  Stage out:4588  Finished in previous run:1253574  Finished successfully:31064
Progress:  time: Mon, 17 Dec 2012 21:04:05 +0000  Selecting site:26509  Stage in:5232  Submitting:3  Active:4230  Stage out:4653  Finished in previous run:1253574  Finished successfully:31114
Progress:  time: Mon, 17 Dec 2012 21:04:30 +0000  Selecting site:26470  Stage in:5245  Active:4253  Checking status:1  Stage out:4619  Finished in previous run:1253574  Finished successfully:31153
Progress:  time: Mon, 17 Dec 2012 21:04:31 +0000  Selecting site:26425  Stage in:5274  Submitting:3  Active:4158  Stage out:4683  Finished in previous run:1253574  Finished successfully:31198
Progress:  time: Mon, 17 Dec 2012 21:04:55 +0000  Selecting site:26422  Stage in:5277  Active:4161  Stage out:4680  Finished in previous run:1253574  Finished successfully:31201



From nbest at ci.uchicago.edu  Mon Dec 17 15:33:33 2012
From: nbest at ci.uchicago.edu (Neil Best)
Date: Mon, 17 Dec 2012 15:33:33 -0600
Subject: [Swift-user] ignore exceptions?
Message-ID: <CANVU8H2Z3+Zbv0t3gjLTb59oMn=M5Rmi-TMc7LFchTX4fxgvXQ@mail.gmail.com>

I have an executable that is throwing exception after it finishes its
work.  I am treating these crashes as benign but Swift regards them as
failures even though the data that it is writing out as a side effect
is valid.  Is there a way to get Swift to ignore these exceptions?
It's a home-cooked Fortran executable and I suspect that its memory
allocation and/or file handles are not being released cleanly or
something of that nature.  I am trying to avoid debugging it as it
would not really be a good use of my time if I don't have to.  Thanks.


From wilde at mcs.anl.gov  Mon Dec 17 15:51:26 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Mon, 17 Dec 2012 15:51:26 -0600 (CST)
Subject: [Swift-user] ignore exceptions?
In-Reply-To: <CANVU8H2Z3+Zbv0t3gjLTb59oMn=M5Rmi-TMc7LFchTX4fxgvXQ@mail.gmail.com>
Message-ID: <785759066.99123.1355781086918.JavaMail.root@zimbra.anl.gov>

Neil, one solution is to wrap your fortran executable in a shell app that catches the SIGSEGV and/or other signals that the program may be generating, and exits with a 0 return code.

Something like:

trap "{ exit 0; }" EXIT SEGILL SIGTRAP SIGABRT SIGSEGV # etc - whatever applies

- Mike

----- Original Message -----
> From: "Neil Best" <nbest at ci.uchicago.edu>
> To: "Michael Wilde" <wilde at mcs.anl.gov>, swift-user at ci.uchicago.edu
> Sent: Monday, December 17, 2012 3:33:33 PM
> Subject: ignore exceptions?
> I have an executable that is throwing exception after it finishes its
> work. I am treating these crashes as benign but Swift regards them as
> failures even though the data that it is writing out as a side effect
> is valid. Is there a way to get Swift to ignore these exceptions?
> It's a home-cooked Fortran executable and I suspect that its memory
> allocation and/or file handles are not being released cleanly or
> something of that nature. I am trying to avoid debugging it as it
> would not really be a good use of my time if I don't have to. Thanks.

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From yuyanglai88 at gmail.com  Mon Dec 17 18:15:18 2012
From: yuyanglai88 at gmail.com (Thomas)
Date: Mon, 17 Dec 2012 16:15:18 -0800
Subject: [Swift-user] Swift error handling
Message-ID: <CAPF5tR84P8FBr_6hExROhpt=kNVDbF83F1gtsm-M1v2VTDDNcA@mail.gmail.com>

Hi,

I noticed that when a bash script that is being called from a Swift script
fails (returns error 1), the Swift script will fail and give a File Not
Found error. Is there an error handling mechanism in Swift that lets us
know if the bash script is failing and lets us handle it in the Swift
script?

Cheers,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20121217/477e5dbc/attachment.html>

From hategan at mcs.anl.gov  Mon Dec 17 18:24:20 2012
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 17 Dec 2012 16:24:20 -0800
Subject: [Swift-user] Swift error handling
In-Reply-To: <CAPF5tR84P8FBr_6hExROhpt=kNVDbF83F1gtsm-M1v2VTDDNcA@mail.gmail.com>
References: <CAPF5tR84P8FBr_6hExROhpt=kNVDbF83F1gtsm-M1v2VTDDNcA@mail.gmail.com>
Message-ID: <1355790260.29026.4.camel@blabla>

There is no swift script level error handling at this time.

The swift runtime may (depending on settings) retry failed applications.

There is always the possibility of wrapping an application in a shell
script that deals with errors from the actual application.

As a side note, the FileNotFound error, should not be immediately
visible to the user, but it may appear in the logs. If that's not the
case, it's probably a bug.

Mihael

On Mon, 2012-12-17 at 16:15 -0800, Thomas wrote:
> Hi,
> 
> 
> I noticed that when a bash script that is being called from a Swift
> script fails (returns error 1), the Swift script will fail and give a
> File Not Found error. Is there an error handling mechanism in Swift
> that lets us know if the bash script is failing and lets us handle it
> in the Swift script?
> 
> 
> Cheers,
> Thomas
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user




From wilde at mcs.anl.gov  Mon Dec 17 18:25:26 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Mon, 17 Dec 2012 18:25:26 -0600 (CST)
Subject: [Swift-user] Swift error handling
In-Reply-To: <CAPF5tR84P8FBr_6hExROhpt=kNVDbF83F1gtsm-M1v2VTDDNcA@mail.gmail.com>
Message-ID: <347188453.99493.1355790326127.JavaMail.root@zimbra.anl.gov>

Hi Thomas,

> I noticed that when a bash script that is being called from a Swift
> script fails (returns error 1), the Swift script will fail and give a
> File Not Found error.

Right. Its complaining that the bash script did not produce one or more files that it was expected to produce based on the app() declaration.

> Is there an error handling mechanism in Swift
> that lets us know if the bash script is failing and lets us handle it
> in the Swift script?

No, not at the present time.

The best way to do this is to check for failures in the bash script and return some representation of that failure in a file, which the Swift script can then process via readData().

for example:

(out, error) = myBashScript(args);
errorString = readData(error);
if (errorString == "SomeErrorCondition") {
  # handle error here...
}

- Mike

> Cheers,
> Thomas

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From nbest at ci.uchicago.edu  Thu Dec 20 10:24:20 2012
From: nbest at ci.uchicago.edu (Neil Best)
Date: Thu, 20 Dec 2012 10:24:20 -0600
Subject: [Swift-user] swift on midway: apps and modules
In-Reply-To: <1415331725.68060.1354580546585.JavaMail.root@zimbra.anl.gov>
References: <CANVU8H3om0Gizxdh185gAX63+hWJ89CzqJ=pJiiz3KZ6TVD6xQ@mail.gmail.com>
	<1415331725.68060.1354580546585.JavaMail.root@zimbra.anl.gov>
Message-ID: <CANVU8H06nq_X1HaatCdVxmRB7DnqAwYvsh5H3UuU4V8gGYEiQA@mail.gmail.com>

On Mon, Dec 3, 2012 at 6:22 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

>
>   use.provider.staging=true
>
> Now your input data files will be copied to the node /scratch/local dir and your output files copied back.
>


I assume that coaster staging will only work if the inputs are
explicitly mapped, right?  My app works by giving a path as a
parameter.  The script fired by the app finds its inputs there.  I
guess I need some kind of spoof in that case.


From wilde at mcs.anl.gov  Thu Dec 20 11:07:51 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 20 Dec 2012 11:07:51 -0600 (CST)
Subject: [Swift-user] swift on midway: apps and modules
In-Reply-To: <CANVU8H06nq_X1HaatCdVxmRB7DnqAwYvsh5H3UuU4V8gGYEiQA@mail.gmail.com>
Message-ID: <166633021.105415.1356023271372.JavaMail.root@zimbra.anl.gov>

 
> I assume that coaster staging will only work if the inputs are
> explicitly mapped, right?

Thats right. The files to be staged must be mapped using standard Swift mapping constructs. Files that are accessed by passing strings will not be seen by Swift and hence not staged by provider staging (or any other automatic Swift staging).

> My app works by giving a path as a
> parameter. The script fired by the app finds its inputs there. I
> guess I need some kind of spoof in that case.

You'll need to map all files to take advantage of provider staging.

- Mike


From marialemos72 at gmail.com  Wed Dec 26 14:08:04 2012
From: marialemos72 at gmail.com (Maria Lemos)
Date: Wed, 26 Dec 2012 20:08:04 +0000
Subject: [Swift-user] CISTI'2013 Doctoral Symposium - CFP, Lisbon,
	June 19 - 23, 2013
Message-ID: <20121226200707.ED68B7CC0BA@mailrelay.anl.gov>


***************************************************************************************************
CISTI'2013 DOCTORAL SYMPOSIUM
8th Iberian Conference on Information Systems and Technologies
Lisbon, Portugal, June 19 - 23, 2013
http://www.aisti.eu/cisti2013/index.php?option=com_content&view=article&id=64&Itemid=68&lang=en
***************************************************************************************************

INTRODUCTION

The purpose of CISTI'2013?s Doctoral Symposium (http://www.aisti.eu/cisti2013/index.php?option=com_content&view=article&id=64&Itemid=68&lang=en) is to provide graduate students a setting where they can, informally, expose and discuss their work, collecting valuable expert opinions and sharing new ideas, methods and applications. The Doctoral Symposium is an excellent opportunity for PhD students to present and discuss their work in a Workshop format. Each presentation will be evaluated by a panel composed by at least three Information Systems and Technologies experts. 

CONTRIBUTIONS SUBMISSION

The Doctoral Symposium is opened to PhD students whose research area includes the themes proposed for this Conference. Submissions must include an extended abstract (maximum 4 pages), following the Conference style guide. All selected contributions will be handed out along with the Conference Proceedings, in CD with an ISBN. These contributions will be send for indexation by EBSCO, and EI-Compendex.

Submissions must include the field, the PhD institution and the number of months devoted to the development of the work. Additionally, they should include in a clear and succinct manner:

    ?    The problem approached and its significance or relevance
    ?    The research objectives and related investigation topics
    ?    A brief display of what is already known
    ?    A proposed solution methodology for the problem
    ?    Expected results

IMPORTANT DATES

?    Data limite para submiss?o de propostas: 15 de Fevereiro de 2013
?    Notifica??o de aceita??o: 29 de Mar?o de 2013
?    Data limite para apresenta??o das vers?es finais: 12 de Abril de 2013
?    Pagamento da inscri??o, para garantir a inclus?o da contribui??o aceite nas actas da confer?ncia: 12 de Abril de 2013

SCIENTIFIC AND ORGANIZING COMMITTEE

Manuel P?rez Cota, Universidad de Vigo (Chair)
Adolfo Lozano Tello, Universidad de Extremadura
Alberto J. Bugar?n Diz, Universidad de Santiago de Compostela
?lvaro Rocha, Universidade Fernando Pessoa
Ana Maria Ramalho Correia, Universidade Nova de Lisboa, ISEGI
Ant?nio Palma dos Reis, Universidade T?cnica de Lisboa, ISEG
Arturo Mendez Pen?n, Universidade de Vigo
Carlos Ferr?s Sexto, Universidad de Santiago de Compostela
David Fonseca, Universidad Ram?n Llul
Ernesto Redondo, Universidad Polit?cnica de Catalu?a
Feliz Gouveia, Universidade Fernando Pessoa
Francisco Restivo, Universidade Cat?lica Portuguesa - Braga
Guilhermina Miranda, Universidade de Lisboa
Gonzalo Cuevas Agust?n, Universidad Polit?cnica de Madrid
H?ctor Jorge Garc?a Neder, Universidad Tecnol?gica NacioNal
Jo?o ?lvaro Carvalho, Universidade do Minho
Jo?o Barroso, Universidade de Tr?s-os-Montes e Alto Douro
J?rg Thomaschewski, University of Applied Sciences of Emden-Leer
Jos? Antonio Calvo-Manzano Villal?n, Universidad Polit?cnica de Madrid
Jos? Bulas Cruz, Universidade de Tr?s-os-Montes e Alto Douro
Jos? Tribolet, Universidade T?cnica de Lisboa, IST
Leandro Rodr?guez Li?ares, Universidade de Vigo
Lu?s Paulo Reis, Universidade do Minho
Mar?a Jos? Lado Touri?o, Universidade de Vigo
Maria Manuela Cruz Cunha, Instituto Polit?cnico do C?vado e do Ave
Marco Painho, Universidade Nova de Lisboa, ISEGI
Mario Alberto Groppo, Universidad Tecnol?gica Nacional
Nuno Ribeiro, Universidade Fernando Pessoa
Pilar Mareca, Universidade Polit?cnica de Madrid
Ramiro Gon?alves, Universidade de Tr?s-os-Montes e Alto Douro
Tomas San Feliu Gilabert, Universidad Polit?cnica de Madrid
Vicente Alcober, Universidad Polit?cnica de Madrid

-
CISTI'2013 Team
http://www.aisti.eu/cisti2013
aistic at gmail.com

-------------------