[Swift-user] Output Files, ReadData and Order of Execution
Carolyn Phillips
cphillips at mcs.anl.gov
Sun Sep 2 22:16:05 CDT 2012
You are right. That was my problem. (dumb!)
Anyway. My next issue is that Swift is telling me it can't find a file that exists. Perhaps this is because it does not understand absolute directory paths the way I am specifying them?
The short version is that, using a simple mapper, I specify the location as ;location="//scratch/midway/phillicl/SwiftJobs/" Note that I had to put two // at the beginning because the first backslash gets removed for some reason. Then I pass that file name to a script write to that file. But then Swift doesn't see the file
Here is the more detailed version
I have a script called launch jobs that does the following
> cd /scratch/midway/phillicl/SwiftJobs
>
> mkdir Job.${1}.${2}
> cd Job.${1}.${2}
>
> # Copy in some files and do some work
>
> pwd > ${3}
Here is the swift script
> # Types
> type file;
> type unlabeleddata;
> type labeleddata;
> type errorlog;
>
> # Structured Types
> type pointfile {
> unlabeleddata points;
> errorlog error;
> }
>
> type simulationfile {
> file output;
> }
>
> # Apps
> app (file o) cat (file i)
> {
> cat @i stdout=@o;
> }
>
> app (file o) cat2 (file i)
> {
> systeminfo stdout=@o;
> }
>
> app (pointfile o) generatepoints (file c, labeleddata f, string mode, int Npoints)
> {
> matlab_callgeneratepoints @c @f mode Npoints @o.points @o.error;
> }
>
> app (simulationfile o) runSimulation(string p,int passes, int pindex)
> {
> launchjob passes pindex @o.output;
> }
>
> #Files (using single file mapper)
> file config <"designspace.config">;
> labeleddata labeledpoints <"emptypoints.dat">;
>
> type pointlog;
>
> # Loop
> iterate passes {
>
> # Generate Parameters
> pointfile np <simple_mapper;prefix="mypoints.",suffix=".dat">;
> np = generatepoints(config,labeledpoints, "uniform", 50);
>
> errorlog fe = np.error;
> int checkforerror = readData(fe);
> tracef("%s: %i\n", "Generate Parameters Error Value", checkforerror);
>
> # Issue Jobs
> simulationfile simfiles[] <simple_mapper;location="//scratch/midway/phillicl/SwiftJobs/",prefix=@strcat(passes,"."),suffix=".job">;
> if(checkforerror==0) {
> unlabeleddata pl = np.points;
> string parameters[] =readData(pl);
> foreach p,pindex in parameters {
> tracef("Launch Job for Parameters: %s\n", p);
> simfiles[pindex] = runSimulation(p,passes,pindex);
> }
> }
>
> # Analyze Jobs
>
> # Generate Prediction
>
>
>
> # creates an array of datafiles named swifttest.<passes>.out to write to
> file out[]<simple_mapper; location=".", prefix=@strcat("swifttest.",passes,"."),suffix=".out">;
>
> # creates a default of 10 files
> foreach j in [1:@toInt(@arg("n","10"))] {
> file data<"data.txt">;
> out[j] = cat2(data);
> }
>
> # try writing the iteration to a log file
> file passlog <"passes.log">;
> passlog = writeData(passes);
>
> # try reading from another log file
> int readpasses = readData(passlog);
>
> # Write to the Output Log
> tracef("%s: %i\n", "Iteration :", passes);
> tracef("%s: %i\n", "Iteration Read :", readpasses);
>
> #} until (readpasses == 2); # Determine if Done
> } until (passes == 1); # Determine if Done
And Here is the error I get:
EXCEPTION Exception in launchjob:
Arguments: [0, 1, /scratch/midway/phillicl/SwiftJobs/0.0001.output.job]
Host: pbs
Directory: test-20120903-0303-pedfpqu8/jobs/f/launchjob-fff1mjxk
stderr.txt:
stdout.txt:
----
sys:exception @ vdl-int.k, line: 601
sys:throw @ vdl-int.k, line: 600
sys:catch @ vdl-int.k, line: 567
sys:try @ vdl-int.k, line: 469
task:allocatehost @ vdl-int.k, line: 419
vdl:execute2 @ execute-default.k, line: 23
sys:ignoreerrors @ execute-default.k, line: 21
sys:parallelfor @ execute-default.k, line: 20
sys:restartonerror @ execute-default.k, line: 16
sys:sequential @ execute-default.k, line: 14
sys:try @ execute-default.k, line: 13
sys:if @ execute-default.k, line: 12
sys:then @ execute-default.k, line: 11
sys:if @ execute-default.k, line: 10
vdl:execute @ test.kml, line: 182
run_simulation @ test.kml, line: 480
sys:parallel @ test.kml, line: 465
foreach @ test.kml, line: 456
sys:parallel @ test.kml, line: 427
sys:then @ test.kml, line: 409
sys:if @ test.kml, line: 404
sys:sequential @ test.kml, line: 402
sys:parallel @ test.kml, line: 315
iterate @ test.kml, line: 229
vdl:sequentialwithid @ test.kml, line: 226
vdl:mainp @ test.kml, line: 225
mainp @ vdl.k, line: 118
vdl:mains @ test.kml, line: 223
vdl:mains @ test.kml, line: 223
rlog:restartlog @ test.kml, line: 222
kernel:project @ test.kml, line: 2
test-20120903-0303-pedfpqu8
Caused by: The following output files were not created by the application: /scratch/midway/phillicl/SwiftJobs/0.0001.output.job
Note that
> ls /scratch/midway/phillicl/SwiftJobs/0.0001.output.job
/scratch/midway/phillicl/SwiftJobs/0.0001.output.job
On Sep 1, 2012, at 5:45 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> The error comes from int checkforerror = readData(np.error);
>
> You have to use the workaround for both.
>
> On Sat, 2012-09-01 at 15:23 -0500, Carolyn Phillips wrote:
>> Sure
>>
>> There are a lot of extra stuff running around in the script, fyi
>>
>> # Types
>> type file;
>> type unlabeleddata;
>> type labeleddata;
>> type errorlog;
>>
>> # Structured Types
>> type pointfile {
>> unlabeleddata points;
>> errorlog error;
>> }
>>
>> type simulationfile {
>> file output;
>> }
>>
>> # Apps
>> app (file o) cat (file i)
>> {
>> cat @i stdout=@o;
>> }
>>
>> app (file o) cat2 (file i)
>> {
>> systeminfo stdout=@o;
>> }
>>
>> app (pointfile o) generatepoints (file c, labeleddata f, string mode, int Npoints)
>> {
>> matlab_callgeneratepoints @c @f mode Npoints @o.points @o.error;
>> }
>>
>> #app (simulationfile o) runSimulation(string p)
>> #{
>> # launchjob p @o.output;
>> #}
>>
>> #Files (using single file mapper)
>> file config <"designspace.config">;
>> labeleddata labeledpoints <"emptypoints.dat">;
>>
>> type pointlog;
>>
>> # Loop
>> iterate passes {
>>
>> # Generate Parameters
>> pointfile np <simple_mapper;prefix="mypoints.",suffix=".dat">;
>> np = generatepoints(config,labeledpoints, "uniform", 50);
>>
>> int checkforerror = readData(np.error);
>> tracef("%s: %i\n", "Generate Parameters Error Value", checkforerror);
>>
>> # Issue Jobs
>> #simulationfile simfiles[] <simple_mapper;location="/scratch/midway/phillicl/SwiftJobs/",prefix=@strcat("output.",passes,"."),suffix=".job">;
>> if(checkforerror==0) {
>> unlabeleddata pl = np.points;
>> string parameters[] =readData(pl);
>> foreach p,pindex in parameters {
>> tracef("Launch Job for Parameters: %s\n", p);
>> #simfiles[pindex] = runSimulation(p);
>> }
>> }
>>
>> # Analyze Jobs
>>
>> # Generate Prediction
>>
>>
>>
>> # creates an array of datafiles named swifttest.<passes>.out to write to
>> file out[]<simple_mapper; location=".", prefix=@strcat("swifttest.",passes,"."),suffix=".out">;
>>
>> # creates a default of 10 files
>> foreach j in [1:@toInt(@arg("n","10"))] {
>> file data<"data.txt">;
>> out[j] = cat2(data);
>> }
>>
>> # try writing the iteration to a log file
>> file passlog <"passes.log">;
>> passlog = writeData(passes);
>>
>> # try reading from another log file
>> int readpasses = readData(passlog);
>>
>> # Write to the Output Log
>> tracef("%s: %i\n", "Iteration :", passes);
>> tracef("%s: %i\n", "Iteration Read :", readpasses);
>>
>> #} until (readpasses == 2); # Determine if Done
>> } until (passes == 1); # Determine if Done
>>
>>
>> On Sep 1, 2012, at 1:57 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>
>>> Can you post the entire script?
>>>
>>> On Sat, 2012-09-01 at 12:29 -0500, Carolyn Phillips wrote:
>>>> Yes, I tried that
>>>>
>>>> unlabeleddata pl = np.points;
>>>> string parameters[] =readData(pl);
>>>>
>>>>
>>>> and I got
>>>>
>>>> Execution failed:
>>>> mypoints..dat (No such file or directory)
>>>>
>>>> On Aug 31, 2012, at 8:27 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>>>
>>>>> On Fri, 2012-08-31 at 20:11 -0500, Carolyn Phillips wrote:
>>>>>> How would this line work for what I have below?
>>>>>>
>>>>>>>> string parameters[] =readData(np.points);
>>>>>>
>>>>>
>>>>> unlabeleddata tmp = np.points;
>>>>> string parameters[] = readData(tmp);
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Aug 31, 2012, at 7:49 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> Another bug.
>>>>>>>
>>>>>>> I committed a fix. In the mean time, the solution is:
>>>>>>>
>>>>>>>
>>>>>>> errorlog fe = np.errorlog;
>>>>>>>
>>>>>>> int error = readData(fe);
>>>>>>>
>>>>>>> On Fri, 2012-08-31 at 19:29 -0500, Carolyn Phillips wrote:
>>>>>>>> Hi Mihael,
>>>>>>>>
>>>>>>>> the reason I added the "@" was because
>>>>>>>>
>>>>>>>> now this (similar) line
>>>>>>>>
>>>>>>>> if(checkforerror==0) {
>>>>>>>> string parameters[] =readData(np.points);
>>>>>>>> }
>>>>>>>>
>>>>>>>> gives me this:
>>>>>>>>
>>>>>>>> Execution failed:
>>>>>>>> mypoints..dat (No such file or directory)
>>>>>>>>
>>>>>>>> as in now its not getting the name of the file correct
>>>>>>>>
>>>>>>>> On Aug 31, 2012, at 7:17 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>>>>>>>
>>>>>>>>> @np.error means the file name of np.error which is known statically. So
>>>>>>>>> readData(@np.error) can run as soon as the script starts.
>>>>>>>>>
>>>>>>>>> You probably want to say readData(np.error).
>>>>>>>>>
>>>>>>>>> Mihael
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, 2012-08-31 at 18:55 -0500, Carolyn Phillips wrote:
>>>>>>>>>> So I execute an atomic procedure to generate a datafile, and then next
>>>>>>>>>> I want to do something with that data file. However, my program is
>>>>>>>>>> trying to do something with the datafile before it has been written
>>>>>>>>>> to. So something with order of execution is not working. I think the
>>>>>>>>>> problem is that the name of my file exists, but the file itself does
>>>>>>>>>> not yet, but execution proceeds anyway!
>>>>>>>>>>
>>>>>>>>>> Here are my lines
>>>>>>>>>>
>>>>>>>>>> type pointfile {
>>>>>>>>>> unlabeleddata points;
>>>>>>>>>> errorlog error;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> # Generate Parameters
>>>>>>>>>> pointfile np <simple_mapper;prefix="mypoints.",suffix=".dat">;
>>>>>>>>>> np = generatepoints(config,labeledpoints, "uniform", 50);
>>>>>>>>>>
>>>>>>>>>> int checkforerror = readData(@np.error);
>>>>>>>>>>
>>>>>>>>>> This gives an error :
>>>>>>>>>> mypoints.error.dat (No such file or directory)
>>>>>>>>>>
>>>>>>>>>> If I comment out the last line.. all the files show up in the directory. (e.g. mypoints.points.dat and mypoints.error.dat) ) and if forget to remove the .dat files from a prior run, it also runs fine!
>>>>>>>>>>
>>>>>>>>>> How do you fix a problem like that?
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Swift-user mailing list
>>>>>>>>>> Swift-user at ci.uchicago.edu
>>>>>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
More information about the Swift-user
mailing list