[Swift-user] Output Files, ReadData and Order of Execution
Carolyn Phillips
cphillips at mcs.anl.gov
Mon Sep 10 13:46:13 CDT 2012
Hey Mike. This worked, but I still had to add the extra "/"
-Carolyn
On Sep 7, 2012, at 2:52 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:
> Carolyn, to follow up on your question: below is an example of using the "direct" file access mode.
>
> - Mike
>
> #----- The swift script
>
> $ cat catsndirect.swift
>
> type file;
>
> app (file o) cat (file i)
> {
> cat @i stdout=@o;
> }
>
> file out[]<simple_mapper; location="/tmp/wilde/outdir", prefix="f.",suffix=".out">;
>
> foreach j in [1:@toint(@arg("n","1"))] {
> file data<"/tmp/wilde/indir/data.txt">;
> out[j] = cat(data);
> }
>
> #----- The "cdm" file:
>
> $ cat direct
> rule .* DIRECT /
>
> #----- The command line:
>
> $ swift -config cf -cdm.file direct -tc.file tc -sites.file sites.xml catsndirect.swift -n=10
>
> #----- The output and input dirs:
>
> $ ls -lr /tmp/wilde/{in,out}dir
>
> /tmp/wilde/outdir:
>
> total 40
> -rw-r--r-- 1 wilde ci-users 8 Sep 7 14:11 f.0010.out
> -rw-r--r-- 1 wilde ci-users 8 Sep 7 14:11 f.0009.out
> -rw-r--r-- 1 wilde ci-users 8 Sep 7 14:11 f.0008.out
> -rw-r--r-- 1 wilde ci-users 8 Sep 7 14:11 f.0007.out
> -rw-r--r-- 1 wilde ci-users 8 Sep 7 14:11 f.0006.out
> -rw-r--r-- 1 wilde ci-users 8 Sep 7 14:11 f.0005.out
> -rw-r--r-- 1 wilde ci-users 8 Sep 7 14:11 f.0004.out
> -rw-r--r-- 1 wilde ci-users 8 Sep 7 14:11 f.0003.out
> -rw-r--r-- 1 wilde ci-users 8 Sep 7 14:11 f.0002.out
> -rw-r--r-- 1 wilde ci-users 8 Sep 7 14:11 f.0001.out
>
> /tmp/wilde/indir:
>
> total 4
> -rw-r--r-- 1 wilde ci-users 8 Sep 7 13:47 data.txt
> com$
>
>
> ----- Original Message -----
>> From: "Michael Wilde" <wilde at mcs.anl.gov>
>> To: "Carolyn Phillips" <cphillips at mcs.anl.gov>
>> Cc: swift-user at ci.uchicago.edu
>> Sent: Wednesday, September 5, 2012 8:40:28 AM
>> Subject: Re: [Swift-user] Output Files, ReadData and Order of Execution
>> Hi Carolyn,
>>
>> I think this error is due to the fact that the launchjob script is not
>> coded to match Swift's file management conventions.
>>
>> Unless you declare that you want to use "Direct" file management,
>> Swift will expect output files to be created relative to the directory
>> in which it runs your app() scripts. Thats why it was pulling off the
>> leading "/". By putting a // at the front of the pathname, you were
>> inadvertantly causing your launchjob script to place its output file
>> in a different directory than where Swift was expecting it.
>>
>> There's a further mismatch I think between the mapped filename from
>> simple_mapper (which defaults to 4-digit strings for indices) and the
>> names that launchjob is trying to create.
>>
>> I'll need to send further clarification later, but for now, could you
>> try the following:
>>
>> - go back to using a single leading "/"
>> - comment out the mkdir and cd in launchjob, as $3 contains the
>> correct pathname to write to (which will be a long relative pathname
>> without the leaning "/")
>>
>> I think what you really want here is "DIRECT" file management mode,
>> explained at:
>>
>> http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_policy_descriptions
>>
>> We need to enhance the User Guide to explain this clearly and fully.
>>
>> - Mike
>>
>> ----- Original Message -----
>>> From: "Carolyn Phillips" <cphillips at mcs.anl.gov>
>>> To: "Mihael Hategan" <hategan at mcs.anl.gov>
>>> Cc: swift-user at ci.uchicago.edu
>>> Sent: Sunday, September 2, 2012 10:16:05 PM
>>> Subject: Re: [Swift-user] Output Files, ReadData and Order of
>>> Execution
>>> You are right. That was my problem. (dumb!)
>>>
>>> Anyway. My next issue is that Swift is telling me it can't find a
>>> file
>>> that exists. Perhaps this is because it does not understand absolute
>>> directory paths the way I am specifying them?
>>>
>>> The short version is that, using a simple mapper, I specify the
>>> location as ;location="//scratch/midway/phillicl/SwiftJobs/" Note
>>> that
>>> I had to put two // at the beginning because the first backslash
>>> gets
>>> removed for some reason. Then I pass that file name to a script
>>> write
>>> to that file. But then Swift doesn't see the file
>>>
>>> Here is the more detailed version
>>>
>>> I have a script called launch jobs that does the following
>>>> cd /scratch/midway/phillicl/SwiftJobs
>>>>
>>>> mkdir Job.${1}.${2}
>>>> cd Job.${1}.${2}
>>>>
>>>> # Copy in some files and do some work
>>>>
>>>> pwd > ${3}
>>>
>>>
>>>
>>> Here is the swift script
>>>
>>>> # Types
>>>> type file;
>>>> type unlabeleddata;
>>>> type labeleddata;
>>>> type errorlog;
>>>>
>>>> # Structured Types
>>>> type pointfile {
>>>> unlabeleddata points;
>>>> errorlog error;
>>>> }
>>>>
>>>> type simulationfile {
>>>> file output;
>>>> }
>>>>
>>>> # Apps
>>>> app (file o) cat (file i)
>>>> {
>>>> cat @i stdout=@o;
>>>> }
>>>>
>>>> app (file o) cat2 (file i)
>>>> {
>>>> systeminfo stdout=@o;
>>>> }
>>>>
>>>> app (pointfile o) generatepoints (file c, labeleddata f, string
>>>> mode, int Npoints)
>>>> {
>>>> matlab_callgeneratepoints @c @f mode Npoints @o.points @o.error;
>>>> }
>>>>
>>>> app (simulationfile o) runSimulation(string p,int passes, int
>>>> pindex)
>>>> {
>>>> launchjob passes pindex @o.output;
>>>> }
>>>>
>>>> #Files (using single file mapper)
>>>> file config <"designspace.config">;
>>>> labeleddata labeledpoints <"emptypoints.dat">;
>>>>
>>>> type pointlog;
>>>>
>>>> # Loop
>>>> iterate passes {
>>>>
>>>> # Generate Parameters
>>>> pointfile np <simple_mapper;prefix="mypoints.",suffix=".dat">;
>>>> np = generatepoints(config,labeledpoints, "uniform", 50);
>>>>
>>>> errorlog fe = np.error;
>>>> int checkforerror = readData(fe);
>>>> tracef("%s: %i\n", "Generate Parameters Error Value",
>>>> checkforerror);
>>>>
>>>> # Issue Jobs
>>>> simulationfile simfiles[]
>>>> <simple_mapper;location="//scratch/midway/phillicl/SwiftJobs/",prefix=@strcat(passes,"."),suffix=".job">;
>>>> if(checkforerror==0) {
>>>> unlabeleddata pl = np.points;
>>>> string parameters[] =readData(pl);
>>>> foreach p,pindex in parameters {
>>>> tracef("Launch Job for Parameters: %s\n", p);
>>>> simfiles[pindex] = runSimulation(p,passes,pindex);
>>>> }
>>>> }
>>>>
>>>> # Analyze Jobs
>>>>
>>>> # Generate Prediction
>>>>
>>>>
>>>>
>>>> # creates an array of datafiles named swifttest.<passes>.out
>>>> to
>>>> write to
>>>> file out[]<simple_mapper; location=".",
>>>> prefix=@strcat("swifttest.",passes,"."),suffix=".out">;
>>>>
>>>> # creates a default of 10 files
>>>> foreach j in [1:@toInt(@arg("n","10"))] {
>>>> file data<"data.txt">;
>>>> out[j] = cat2(data);
>>>> }
>>>>
>>>> # try writing the iteration to a log file
>>>> file passlog <"passes.log">;
>>>> passlog = writeData(passes);
>>>>
>>>> # try reading from another log file
>>>> int readpasses = readData(passlog);
>>>>
>>>> # Write to the Output Log
>>>> tracef("%s: %i\n", "Iteration :", passes);
>>>> tracef("%s: %i\n", "Iteration Read :", readpasses);
>>>>
>>>> #} until (readpasses == 2); # Determine if Done
>>>> } until (passes == 1); # Determine if Done
>>>
>>>
>>> And Here is the error I get:
>>>
>>> EXCEPTION Exception in launchjob:
>>> Arguments: [0, 1,
>>> /scratch/midway/phillicl/SwiftJobs/0.0001.output.job]
>>> Host: pbs
>>> Directory: test-20120903-0303-pedfpqu8/jobs/f/launchjob-fff1mjxk
>>> stderr.txt:
>>> stdout.txt:
>>> ----
>>>
>>> sys:exception @ vdl-int.k, line: 601
>>> sys:throw @ vdl-int.k, line: 600
>>> sys:catch @ vdl-int.k, line: 567
>>> sys:try @ vdl-int.k, line: 469
>>> task:allocatehost @ vdl-int.k, line: 419
>>> vdl:execute2 @ execute-default.k, line: 23
>>> sys:ignoreerrors @ execute-default.k, line: 21
>>> sys:parallelfor @ execute-default.k, line: 20
>>> sys:restartonerror @ execute-default.k, line: 16
>>> sys:sequential @ execute-default.k, line: 14
>>> sys:try @ execute-default.k, line: 13
>>> sys:if @ execute-default.k, line: 12
>>> sys:then @ execute-default.k, line: 11
>>> sys:if @ execute-default.k, line: 10
>>> vdl:execute @ test.kml, line: 182
>>> run_simulation @ test.kml, line: 480
>>> sys:parallel @ test.kml, line: 465
>>> foreach @ test.kml, line: 456
>>> sys:parallel @ test.kml, line: 427
>>> sys:then @ test.kml, line: 409
>>> sys:if @ test.kml, line: 404
>>> sys:sequential @ test.kml, line: 402
>>> sys:parallel @ test.kml, line: 315
>>> iterate @ test.kml, line: 229
>>> vdl:sequentialwithid @ test.kml, line: 226
>>> vdl:mainp @ test.kml, line: 225
>>> mainp @ vdl.k, line: 118
>>> vdl:mains @ test.kml, line: 223
>>> vdl:mains @ test.kml, line: 223
>>> rlog:restartlog @ test.kml, line: 222
>>> kernel:project @ test.kml, line: 2
>>> test-20120903-0303-pedfpqu8
>>> Caused by: The following output files were not created by the
>>> application: /scratch/midway/phillicl/SwiftJobs/0.0001.output.job
>>>
>>> Note that
>>>> ls /scratch/midway/phillicl/SwiftJobs/0.0001.output.job
>>> /scratch/midway/phillicl/SwiftJobs/0.0001.output.job
>>>
>>>
>>> On Sep 1, 2012, at 5:45 PM, Mihael Hategan <hategan at mcs.anl.gov>
>>> wrote:
>>>
>>>> The error comes from int checkforerror = readData(np.error);
>>>>
>>>> You have to use the workaround for both.
>>>>
>>>> On Sat, 2012-09-01 at 15:23 -0500, Carolyn Phillips wrote:
>>>>> Sure
>>>>>
>>>>> There are a lot of extra stuff running around in the script, fyi
>>>>>
>>>>> # Types
>>>>> type file;
>>>>> type unlabeleddata;
>>>>> type labeleddata;
>>>>> type errorlog;
>>>>>
>>>>> # Structured Types
>>>>> type pointfile {
>>>>> unlabeleddata points;
>>>>> errorlog error;
>>>>> }
>>>>>
>>>>> type simulationfile {
>>>>> file output;
>>>>> }
>>>>>
>>>>> # Apps
>>>>> app (file o) cat (file i)
>>>>> {
>>>>> cat @i stdout=@o;
>>>>> }
>>>>>
>>>>> app (file o) cat2 (file i)
>>>>> {
>>>>> systeminfo stdout=@o;
>>>>> }
>>>>>
>>>>> app (pointfile o) generatepoints (file c, labeleddata f, string
>>>>> mode, int Npoints)
>>>>> {
>>>>> matlab_callgeneratepoints @c @f mode Npoints @o.points @o.error;
>>>>> }
>>>>>
>>>>> #app (simulationfile o) runSimulation(string p)
>>>>> #{
>>>>> # launchjob p @o.output;
>>>>> #}
>>>>>
>>>>> #Files (using single file mapper)
>>>>> file config <"designspace.config">;
>>>>> labeleddata labeledpoints <"emptypoints.dat">;
>>>>>
>>>>> type pointlog;
>>>>>
>>>>> # Loop
>>>>> iterate passes {
>>>>>
>>>>> # Generate Parameters
>>>>> pointfile np <simple_mapper;prefix="mypoints.",suffix=".dat">;
>>>>> np = generatepoints(config,labeledpoints, "uniform", 50);
>>>>>
>>>>> int checkforerror = readData(np.error);
>>>>> tracef("%s: %i\n", "Generate Parameters Error Value",
>>>>> checkforerror);
>>>>>
>>>>> # Issue Jobs
>>>>> #simulationfile simfiles[]
>>>>> <simple_mapper;location="/scratch/midway/phillicl/SwiftJobs/",prefix=@strcat("output.",passes,"."),suffix=".job">;
>>>>> if(checkforerror==0) {
>>>>> unlabeleddata pl = np.points;
>>>>> string parameters[] =readData(pl);
>>>>> foreach p,pindex in parameters {
>>>>> tracef("Launch Job for Parameters: %s\n", p);
>>>>> #simfiles[pindex] = runSimulation(p);
>>>>> }
>>>>> }
>>>>>
>>>>> # Analyze Jobs
>>>>>
>>>>> # Generate Prediction
>>>>>
>>>>>
>>>>>
>>>>> # creates an array of datafiles named swifttest.<passes>.out
>>>>> to
>>>>> write to
>>>>> file out[]<simple_mapper; location=".",
>>>>> prefix=@strcat("swifttest.",passes,"."),suffix=".out">;
>>>>>
>>>>> # creates a default of 10 files
>>>>> foreach j in [1:@toInt(@arg("n","10"))] {
>>>>> file data<"data.txt">;
>>>>> out[j] = cat2(data);
>>>>> }
>>>>>
>>>>> # try writing the iteration to a log file
>>>>> file passlog <"passes.log">;
>>>>> passlog = writeData(passes);
>>>>>
>>>>> # try reading from another log file
>>>>> int readpasses = readData(passlog);
>>>>>
>>>>> # Write to the Output Log
>>>>> tracef("%s: %i\n", "Iteration :", passes);
>>>>> tracef("%s: %i\n", "Iteration Read :", readpasses);
>>>>>
>>>>> #} until (readpasses == 2); # Determine if Done
>>>>> } until (passes == 1); # Determine if Done
>>>>>
>>>>>
>>>>> On Sep 1, 2012, at 1:57 PM, Mihael Hategan <hategan at mcs.anl.gov>
>>>>> wrote:
>>>>>
>>>>>> Can you post the entire script?
>>>>>>
>>>>>> On Sat, 2012-09-01 at 12:29 -0500, Carolyn Phillips wrote:
>>>>>>> Yes, I tried that
>>>>>>>
>>>>>>> unlabeleddata pl = np.points;
>>>>>>> string parameters[] =readData(pl);
>>>>>>>
>>>>>>>
>>>>>>> and I got
>>>>>>>
>>>>>>> Execution failed:
>>>>>>> mypoints..dat (No such file or directory)
>>>>>>>
>>>>>>> On Aug 31, 2012, at 8:27 PM, Mihael Hategan
>>>>>>> <hategan at mcs.anl.gov>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Fri, 2012-08-31 at 20:11 -0500, Carolyn Phillips wrote:
>>>>>>>>> How would this line work for what I have below?
>>>>>>>>>
>>>>>>>>>>> string parameters[] =readData(np.points);
>>>>>>>>>
>>>>>>>>
>>>>>>>> unlabeleddata tmp = np.points;
>>>>>>>> string parameters[] = readData(tmp);
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Aug 31, 2012, at 7:49 PM, Mihael Hategan
>>>>>>>>> <hategan at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Another bug.
>>>>>>>>>>
>>>>>>>>>> I committed a fix. In the mean time, the solution is:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> errorlog fe = np.errorlog;
>>>>>>>>>>
>>>>>>>>>> int error = readData(fe);
>>>>>>>>>>
>>>>>>>>>> On Fri, 2012-08-31 at 19:29 -0500, Carolyn Phillips wrote:
>>>>>>>>>>> Hi Mihael,
>>>>>>>>>>>
>>>>>>>>>>> the reason I added the "@" was because
>>>>>>>>>>>
>>>>>>>>>>> now this (similar) line
>>>>>>>>>>>
>>>>>>>>>>> if(checkforerror==0) {
>>>>>>>>>>> string parameters[] =readData(np.points);
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> gives me this:
>>>>>>>>>>>
>>>>>>>>>>> Execution failed:
>>>>>>>>>>> mypoints..dat (No such file or directory)
>>>>>>>>>>>
>>>>>>>>>>> as in now its not getting the name of the file correct
>>>>>>>>>>>
>>>>>>>>>>> On Aug 31, 2012, at 7:17 PM, Mihael Hategan
>>>>>>>>>>> <hategan at mcs.anl.gov> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> @np.error means the file name of np.error which is known
>>>>>>>>>>>> statically. So
>>>>>>>>>>>> readData(@np.error) can run as soon as the script starts.
>>>>>>>>>>>>
>>>>>>>>>>>> You probably want to say readData(np.error).
>>>>>>>>>>>>
>>>>>>>>>>>> Mihael
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, 2012-08-31 at 18:55 -0500, Carolyn Phillips wrote:
>>>>>>>>>>>>> So I execute an atomic procedure to generate a datafile,
>>>>>>>>>>>>> and then next
>>>>>>>>>>>>> I want to do something with that data file. However, my
>>>>>>>>>>>>> program is
>>>>>>>>>>>>> trying to do something with the datafile before it has
>>>>>>>>>>>>> been
>>>>>>>>>>>>> written
>>>>>>>>>>>>> to. So something with order of execution is not working.
>>>>>>>>>>>>> I
>>>>>>>>>>>>> think the
>>>>>>>>>>>>> problem is that the name of my file exists, but the file
>>>>>>>>>>>>> itself does
>>>>>>>>>>>>> not yet, but execution proceeds anyway!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here are my lines
>>>>>>>>>>>>>
>>>>>>>>>>>>> type pointfile {
>>>>>>>>>>>>> unlabeleddata points;
>>>>>>>>>>>>> errorlog error;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> # Generate Parameters
>>>>>>>>>>>>> pointfile np
>>>>>>>>>>>>> <simple_mapper;prefix="mypoints.",suffix=".dat">;
>>>>>>>>>>>>> np = generatepoints(config,labeledpoints, "uniform", 50);
>>>>>>>>>>>>>
>>>>>>>>>>>>> int checkforerror = readData(@np.error);
>>>>>>>>>>>>>
>>>>>>>>>>>>> This gives an error :
>>>>>>>>>>>>> mypoints.error.dat (No such file or directory)
>>>>>>>>>>>>>
>>>>>>>>>>>>> If I comment out the last line.. all the files show up in
>>>>>>>>>>>>> the directory. (e.g. mypoints.points.dat and
>>>>>>>>>>>>> mypoints.error.dat) ) and if forget to remove the .dat
>>>>>>>>>>>>> files from a prior run, it also runs fine!
>>>>>>>>>>>>>
>>>>>>>>>>>>> How do you fix a problem like that?
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Swift-user mailing list
>>>>>>>>>>>>> Swift-user at ci.uchicago.edu
>>>>>>>>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>
>> --
>> Michael Wilde
>> Computation Institute, University of Chicago
>> Mathematics and Computer Science Division
>> Argonne National Laboratory
>>
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
More information about the Swift-user
mailing list