[Swift-user] output file array
Ketan Maheshwari
ketan at mcs.anl.gov
Thu Jan 8 10:53:37 CST 2015
On Tue, Jan 6, 2015 at 1:13 PM, Michael Wilde <wilde at anl.gov> wrote:
>
> On 1/6/15 1:04 PM, Ketan Maheshwari wrote:
>
> Hi Mike,
>
> The app is creating the file but the way the app is invoked, the
> filename does not appear in the command-line. It creates the file at the
> top level, so just afile.txt.
>
> However, since there are many calls to the app, to avoid this file being
> overwritten, I need to put this file into a separate directory which is why
> I am using directory outdirN.
>
> I was thinking if it is possible for Swift runtime to find from the app
> definition that the file _appout is expected output similar to stdout and
> move the file to the outdirN (again similar to stdout).
>
> I think what you're asking for is the ability to declare for each app that
> its temporary "sandbox" working dir gets saved below the current working
> dir in which you're running the swift command. And, further, to be able to
> name that directory from the source script.
>
> That *might* be a reasonable feature, but will need more discussion. I
> suggest a bugzilla ticket to capture this as a proposed enhancement. It
> seems however that the current way of doing this, explicitly, is simple and
> sufficient for now.
>
I think it is simpler. Consider the following app:
(file _appout) app someapp (string _appin){
appcmd _appin;
}
file appout <"appout.txt">;
(appout) = someapp ("hello");
Swift will bring the "appout.txt" from workdir to the current dir if it is
produced by app (even without being specified in commandline).
So, I am facing a special case of this pattern where the appout file is
mapped into a directory instead of at the toplevel:
file appout <single_file_mapper; file=strcat("outdir", i, "/appout.txt")>;
Currently, this pattern gets honored for stdout/stderr files, ie. Swift
creates the outdirN and puts the stdout/stderr files into it.
> - Mike
>
>
> Thanks,
> Ketan
>
> On Tue, Jan 6, 2015 at 12:53 PM, Michael Wilde <wilde at anl.gov> wrote:
>
>> Is your app touch_app( ) correctly creating output files of the form
>> outdirN/afile.txt?
>>
>> From the error message, I suspect that it is not.
>>
>> Your app is declared as:
>>
>> app (file _stdout, file _stderr, file _appout) touch_app(string _instr){
>> t _instr stdout=@_stdout stderr=@_stderr;
>> }
>>
>>
>> You need to pass filename(_appout) to the app, via its command line, so
>> that it knows the correct output filename to create. Then you need to
>> ensure that the app does indeed create that file.
>>
>> - Mike
>>
>>
>> On 1/6/15 11:23 AM, Ketan Maheshwari wrote:
>>
>> Trying trunk for this pattern.
>>
>> A toy application invoked over a foreach loop that creates an output
>> file, an stdout and an stderr files.
>>
>> The files are mapped into an output directory named with the loop index
>> as suffix so that the files do not get overwritten:
>>
>> foreach i in [0:9]{
>>
>> file out<single_file_mapper; file=strcat("outdir", i, "/std.out")>;
>> file err<single_file_mapper; file=strcat("outdir", i, "/std.err")>;
>> file appout<single_file_mapper; file=strcat("outdir", i, "/afile.txt")>;
>>
>> (out, err, appout) = touch_app("Hello");
>> }
>>
>>
>> The stdout and stderr files correctly ends up in their respective
>> directories but the app generated file does not.
>>
>> I see following error message:
>>
>> Execution failed:
>> Exception in t:
>> Arguments: [Hello]
>> Host: edison1
>> Directory: touchafile-run001/jobs/t/t-ffv33r2m
>> exception @ swift-int-staging.k, line: 165
>> Caused by: The following output files were not created by the
>> application: outdir4/afile.txt
>>
>> Any suggestions for fixing this?
>>
>> Attached is the test directory with sources and executable with rundir.
>>
>> Thanks,
>> Ketan
>>
>>
>> On Mon, Dec 22, 2014 at 5:14 PM, Mihael Hategan <hategan at mcs.anl.gov>
>> wrote:
>>
>>> Hi,
>>>
>>> I don't think 0.95 supports dynamic arrays output from apps. You will
>>> need trunk/0.96 for that.
>>>
>>> Mihael
>>>
>>> On Mon, 2014-12-22 at 15:37 -0600, Ketan Maheshwari wrote:
>>> > Hi Mihael,
>>> >
>>> > This is with Swift 0.95.
>>> >
>>> > Thanks,
>>> > Ketan
>>> >
>>> > On Sun, Dec 21, 2014 at 2:40 PM, Hategan-Marandiuc, Philip M. <
>>> > hategan at mcs.anl.gov> wrote:
>>> >
>>> > > Hi Ketan,
>>> > >
>>> > > Sorry for the delay. Is this trunk or 0.95?
>>> > >
>>> > > Mihael
>>> > >
>>> > > On Wed, 2014-12-17 at 14:26 -0600, Ketan Maheshwari wrote:
>>> > > > Hi,
>>> > > >
>>> > > > I am dealing with a workflow pattern where an app expects multiple
>>> output
>>> > > > files with a pattern.
>>> > > >
>>> > > > The app signature is:
>>> > > >
>>> > > > app (file[] _wrfout, file _out, file _err) wrf_app (file _wrf_in,
>>> file[]
>>> > > > _tbl, file[] _ozone, ...)
>>> > > > {
>>> > > > wrf stdout=@_out stderr=@_err;
>>> > > > }
>>> > > >
>>> > > > The _wrfout files are the app result files which follows a pattern:
>>> > > wrfout_*
>>> > > >
>>> > > > So, I am invoking the application in a foreach loop as:
>>> > > >
>>> > > > foreach i in [0:2]{
>>> > > > file[] wrfout<simple_mapper; location=strcat("outdir",i),
>>> > > > pattern="wrfout_*">;
>>> > > > file wrfstdout<single_file_mapper; file=strcat("outdir", i,
>>> > > "/std.out")>;
>>> > > > file wrfstderr<single_file_mapper; file=strcat("outdir", i,
>>> > > "/std.err")>;
>>> > > >
>>> > > > (wrfout, wrfstdout, wrfstderr) = wrf_app (wrfin, tbl, ozone,
>>> tr, data,
>>> > > > gribmap, namelist, co2_trans, input_sounding);
>>> > > > }
>>> > > >
>>> > > > The script hangs at runtime with the following messages:
>>> > > >
>>> > > > No events in 1s.
>>> > > > Finding dependency loops...
>>> > > >
>>> > > > Waiting threads:
>>> > > > Thread: R-6-0-4, waiting on wrfout (declared on line 50)
>>> > > > swift:stageOut, wf.edison, line 134
>>> > > > swift:execute, wf.edison, line 123
>>> > > > wrf_app, wf.edison, line 242
>>> > > >
>>> > > > Thread: R-6-2-4, waiting on wrfout (declared on line 50)
>>> > > > swift:stageOut, wf.edison, line 134
>>> > > > swift:execute, wf.edison, line 123
>>> > > > wrf_app, wf.edison, line 242
>>> > > >
>>> > > > Thread: R-6-1-4, waiting on wrfout (declared on line 50)
>>> > > > swift:stageOut, wf.edison, line 134
>>> > > > swift:execute, wf.edison, line 123
>>> > > > wrf_app, wf.edison, line 242
>>> > > >
>>> > > > Any suggestions?
>>> > > >
>>> > > > Thanks,
>>> > > > Ketan
>>> > > > _______________________________________________
>>> > > > Swift-user mailing list
>>> > > > Swift-user at ci.uchicago.edu
>>> > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>> > >
>>> > >
>>> > >
>>>
>>>
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>>
>>
>>
>>
>> _______________________________________________
>> Swift-user mailing listSwift-user at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>
>>
>> --
>> Michael Wilde
>> Mathematics and Computer Science Computation Institute
>> Argonne National Laboratory The University of Chicago
>>
>>
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>
>
>
> --
> Michael Wilde
> Mathematics and Computer Science Computation Institute
> Argonne National Laboratory The University of Chicago
>
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20150108/cbd212d3/attachment.html>
More information about the Swift-user
mailing list