[Swift-devel] CDM Tests

Mon Nov 14 20:51:58 CST 2011

I added a Bugzilla ticket about this issue.
 	Justin

On Mon, 14 Nov 2011, Jonathan Monette wrote:

> I think Justin when I asked him that question was because that was how 
> Swift expected everything.  It didn't break how Swift worked and didn't 
> require any modifications to the Swift source to account for this.  I 
> think this was just a case that wasn't thought out fully on what exactly 
> would happen.  I think CDM knows all the information it needs to 
> actually write the data to the output destination, the logic is just not 
> there.  I didn't know of this behavior with CDM output till this past 
> summer.  I always used CDM for input only until this past summer.
>
> On Nov 14, 2011, at 7:17 PM, Ketan Maheshwari wrote:
>
>> One thing I am not very clear about CDM Direct for output is if there is any benefit in having the outputs symlinked from the workdir? Wouldn't the outputs be expected at the final place in users application output destination?
>>
>> On Mon, Nov 14, 2011 at 6:36 PM, Jonathan Monette <jonmon at mcs.anl.gov> wrote:
>> The "do the right thing" is currently broken in CDM for output files.  A wrapper needs to be written currently around the app that updates the intended file and not the symlink in the work directory.  For instance, in the SwiftMontage scripts I wrap all the Montage calls in python scripts that checks to see if I am writing to a symlink, if I am writing to a symlink I write the what the symlink is pointing to instead of the symlink itself.  So unless a wrapper is written to handle the output files, CDM currently does not work properly.  This probably should be documented in the CDM section of the user guide as a caveat of using CDM for output files.
>>
>> On Nov 14, 2011, at 2:47 PM, Michael Wilde wrote:
>>
>>> I think the purpose of these tests was to find a default CDM DIRECT patter that would match all inout and output files and "do the right thing".
>>>
>>> Tests that match specific pathnames are yet another (but different) useful case.
>>>
>>> - Mike
>>> ----- Original Message -----
>>>> From: "Ketan Maheshwari" <ketancmaheshwari at gmail.com>
>>>> To: "Jon Monette" <jonmon at mcs.anl.gov>
>>>> Cc: "Michael Wilde" <wilde at mcs.anl.gov>, "Swift Devel" <swift-devel at ci.uchicago.edu>
>>>> Sent: Monday, November 14, 2011 12:43:13 PM
>>>> Subject: Re: [Swift-devel] CDM Tests
>>>> Yes, this is right. I added a rule for the outdir in the cdm file
>>>> (which was not added earlier) and all runs fails with the message:
>>>> file not found output/out?.txt.
>>>>
>>>>
>>>> On Mon, Nov 14, 2011 at 2:05 PM, Jon Monette < jonmon at mcs.anl.gov >
>>>> wrote:
>>>>
>>>>
>>>> Output files do not work correctly in CDM. CDM creates symlinks to the
>>>> files that are used in the DIRECT CDM directive. When the app outputs
>>>> it overwrites the symlink in the work directory and *doesn't* update
>>>> the original file. So the check to see if the expected output is the
>>>> same as the correct output will fail. How are you checking to see if
>>>> the app correctly ran? Is there an error or is the original file in
>>>> the cwd empty?
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: "Michael Wilde" < wilde at mcs.anl.gov >
>>>> To: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
>>>> Cc: "Swift Devel" < swift-devel at ci.uchicago.edu >
>>>>
>>>>
>>>>
>>>> Sent: Monday, November 14, 2011 1:59:22 PM
>>>> Subject: Re: [Swift-devel] CDM Tests
>>>>
>>>> I meant the command line to the app() program, not the commandline to
>>>> the swift command.
>>>>
>>>> - Mike
>>>>
>>>> ----- Original Message -----
>>>>> From: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
>>>>> To: "Michael Wilde" < wilde at mcs.anl.gov >
>>>>> Cc: "Swift Devel" < swift-devel at ci.uchicago.edu >
>>>>> Sent: Monday, November 14, 2011 11:50:48 AM
>>>>> Subject: Re: [Swift-devel] CDM Tests
>>>>> All commandlines are in a run.sh script batched together. I am
>>>>> writing
>>>>> a README in the testsuite to document this.
>>>>>
>>>>>
>>>>> On Mon, Nov 14, 2011 at 1:46 PM, Michael Wilde < wilde at mcs.anl.gov >
>>>>> wrote:
>>>>>
>>>>>
>>>>> Ketan, in these tests, I thought that the names that the app
>>>>> actually
>>>>> receives on its command line is worth documenting. Does that show
>>>>> anything that the script writer should be aware of?
>>>>>
>>>>>
>>>>> - Mike
>>>>>
>>>>> ----- Original Message -----
>>>>>
>>>>>
>>>>>
>>>>>> From: "Michael Wilde" < wilde at mcs.anl.gov >
>>>>>> To: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
>>>>>> Cc: "Swift Devel" < swift-devel at ci.uchicago.edu >
>>>>>> Sent: Monday, November 14, 2011 11:39:10 AM
>>>>>> Subject: Re: [Swift-devel] CDM Tests
>>>>>> Even though the tests worked in each of these 4 cases, the
>>>>>> question
>>>>>> remains of "did the app see the same path names" and "did copies
>>>>>> get
>>>>>> done"?
>>>>>>
>>>>>> So I think you need to dig deeper and explain the difference
>>>>>> between
>>>>>> what CDM does for the direct case, and how (if at all) the wrapper
>>>>>> mode affects things.
>>>>>>
>>>>>> Its equally possible that CDM overrode the wrapper mode arg. Just
>>>>>> specifying the wrapper mode arg will not, I think, achieve the
>>>>>> elimination of the "copy to shared workflow directory" that
>>>>>> "direct
>>>>>> mode" accomplishes.
>>>>>>
>>>>>> - Mike
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
>>>>>>> To: "Michael Wilde" < wilde at mcs.anl.gov >
>>>>>>> Cc: "Swift Devel" < swift-devel at ci.uchicago.edu >, "Justin M
>>>>>>> Wozniak"
>>>>>>> < wozniak at mcs.anl.gov >
>>>>>>> Sent: Monday, November 14, 2011 11:29:36 AM
>>>>>>> Subject: Re: [Swift-devel] CDM Tests
>>>>>>> Mike,
>>>>>>>
>>>>>>>
>>>>>>> In my initial email I said:
>>>>>>>
>>>>>>>
>>>>>>> ... when specifying the relative option on the config for
>>>>>>> wrapper.invocation.mode property the script works regardless of
>>>>>>> the
>>>>>>> paths of input/output.
>>>>>>>
>>>>>>>
>>>>>>> Which made me conclude that the option specified in the config
>>>>>>> for
>>>>>>> wrapper.invocation.mode property overrides the CDM policies.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 14, 2011 at 1:21 PM, Michael Wilde <
>>>>>>> wilde at mcs.anl.gov
>>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Ketan,
>>>>>>>
>>>>>>> I thought your initial email stated that the 4 cases of relative
>>>>>>> path
>>>>>>> names "worked". But this latter comment indicates that there is
>>>>>>> some
>>>>>>> kind of problem even in the relative cases.
>>>>>>>
>>>>>>> Can you clarify?
>>>>>>>
>>>>>>>
>>>>>>> - Mike
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> To: "Justin M Wozniak" < wozniak at mcs.anl.gov >
>>>>>>>> Cc: "Michael Wilde" < wilde at mcs.anl.gov >, "Swift Devel" <
>>>>>>>> swift-devel at ci.uchicago.edu >
>>>>>>>> Sent: Monday, November 14, 2011 11:01:35 AM
>>>>>>>> Subject: Re: [Swift-devel] CDM Tests
>>>>>>>> Given the CDM's support for relative pathnames, I would expect
>>>>>>>> the
>>>>>>>> case of relative inputs, relative outputs would succeed
>>>>>>>> irrespective
>>>>>>>> of config. However, from the tests, it seems that the config
>>>>>>>> option
>>>>>>>> is
>>>>>>>> overriding CDM directives.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Nov 14, 2011 at 12:44 PM, Justin M Wozniak <
>>>>>>>> wozniak at mcs.anl.gov > wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> CDM DIRECT does not currently support absolute path names. I
>>>>>>>> think
>>>>>>>> we
>>>>>>>> could provide something simple like "" for the field, as you
>>>>>>>> suggest.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, 14 Nov 2011, Michael Wilde wrote:
>>>>>>>>
>>>>>>>>> I would expect that for absolute path mappings, the user
>>>>>>>>> would
>>>>>>>>> specify
>>>>>>>>> either :: or "/" as the path in the CDM DIRECT pattern. In
>>>>>>>>> other
>>>>>>>>> words,
>>>>>>>>> the user is saying "I have mapped the object to an absolute
>>>>>>>>> path
>>>>>>>>> name,
>>>>>>>>> and I want the app to get exactly this name". Hence I think
>>>>>>>>> that
>>>>>>>>> the
>>>>>>>>> best "location" field for a DIRECT rule is null ("").
>>>>>>>>>
>>>>>>>>> Justin, can you clarify what the code expects for this case?
>>>>>>>>>
>>>>>>>>> - Mike
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From: "Jonathan Monette" < jonmon at mcs.anl.gov >
>>>>>>>>>> To: "Michael Wilde" < wilde at mcs.anl.gov >
>>>>>>>>>> Cc: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >,
>>>>>>>>>> "Swift
>>>>>>>>>> Devel" < swift-devel at ci.uchicago.edu >
>>>>>>>>>> Sent: Monday, November 14, 2011 9:13:01 AM
>>>>>>>>>> Subject: Re: [Swift-devel] CDM Tests
>>>>>>>>>> I couldn't tell from the excel(using my phone) but what was
>>>>>>>>>> absolute?
>>>>>>>>>> If I am not mistaken what CDM does is takes the path that
>>>>>>>>>> is
>>>>>>>>>> specified
>>>>>>>>>> in the CDM file(which should be absolute) and appends the
>>>>>>>>>> file
>>>>>>>>>> path
>>>>>>>>>> that is used in the swift script to it(which should be
>>>>>>>>>> relative).
>>>>>>>>>> So
>>>>>>>>>> if they are both absolute I would expect it to fail.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Nov 14, 2011, at 10:56 AM, Michael Wilde <
>>>>>>>>>> wilde at mcs.anl.gov
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks, Ketan - very nice tests and summary document.
>>>>>>>>>>>
>>>>>>>>>>> I should point out for those that didn't open the doc: the
>>>>>>>>>>> 4
>>>>>>>>>>> tests
>>>>>>>>>>> with absolute pathnames are failing.
>>>>>>>>>>>
>>>>>>>>>>> Ketan, can you work with Justin to see if this is a bug,
>>>>>>>>>>> or
>>>>>>>>>>> if
>>>>>>>>>>> the
>>>>>>>>>>> CDM directive needs to be coded differently for absolute
>>>>>>>>>>> paths?
>>>>>>>>>>> Then
>>>>>>>>>>> please test a fix, and as we discussed adapt the tests
>>>>>>>>>>> with
>>>>>>>>>>> annotations to enhance the User Guide section on CDM.
>>>>>>>>>>>
>>>>>>>>>>> - Mike
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>> From: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
>>>>>>>>>>>> To: "Swift Devel" < swift-devel at ci.uchicago.edu >
>>>>>>>>>>>> Sent: Monday, November 14, 2011 8:13:48 AM
>>>>>>>>>>>> Subject: [Swift-devel] CDM Tests
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I had a discussion with Mike about testing the CDM
>>>>>>>>>>>> behavior
>>>>>>>>>>>> for
>>>>>>>>>>>> the
>>>>>>>>>>>> following cases:
>>>>>>>>>>>> 1. full versus relative paths for input
>>>>>>>>>>>> 2. full versus relative paths for output
>>>>>>>>>>>> 3. relative versus absolute option in config property:
>>>>>>>>>>>> wrapper.invocation.mode
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> In this regard, I made 8 tests for all the above
>>>>>>>>>>>> combinations.
>>>>>>>>>>>> I
>>>>>>>>>>>> used
>>>>>>>>>>>> simple local provider in this first set of tests.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> From the tests it seems that when specifying the relative
>>>>>>>>>>>> option
>>>>>>>>>>>> on
>>>>>>>>>>>> the config for wrapper.invocation.mode property the
>>>>>>>>>>>> script
>>>>>>>>>>>> works
>>>>>>>>>>>> regardless of the paths of input/output.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> A detailed result with stdout, and paths to logs can be
>>>>>>>>>>>> found
>>>>>>>>>>>> here:
>>>>>>>>>>>> https://docs.google.com/spreadsheet/ccc?key=0AmvYSwENKFY9dG44V2VjRXJlUmZLNG9saERFeWZDcUE
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The tests are in my CI dir: /home/ketan/cdm_tests
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards, --
>>>>>>>>>>>> Ketan
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Swift-devel mailing list
>>>>>>>>>>>> Swift-devel at ci.uchicago.edu
>>>>>>>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Michael Wilde
>>>>>>>>>>> Computation Institute, University of Chicago
>>>>>>>>>>> Mathematics and Computer Science Division
>>>>>>>>>>> Argonne National Laboratory
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Swift-devel mailing list
>>>>>>>>>>> Swift-devel at ci.uchicago.edu
>>>>>>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Justin M Wozniak
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Swift-devel mailing list
>>>>>>>> Swift-devel at ci.uchicago.edu
>>>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ketan
>>>>>>>
>>>>>>> --
>>>>>>> Michael Wilde
>>>>>>> Computation Institute, University of Chicago
>>>>>>> Mathematics and Computer Science Division
>>>>>>> Argonne National Laboratory
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ketan
>>>>>>
>>>>>> --
>>>>>> Michael Wilde
>>>>>> Computation Institute, University of Chicago
>>>>>> Mathematics and Computer Science Division
>>>>>> Argonne National Laboratory
>>>>>>
>>>>>> _______________________________________________
>>>>>> Swift-devel mailing list
>>>>>> Swift-devel at ci.uchicago.edu
>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>>
>>>>> --
>>>>> Michael Wilde
>>>>> Computation Institute, University of Chicago
>>>>> Mathematics and Computer Science Division
>>>>> Argonne National Laboratory
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ketan
>>>>
>>>> --
>>>> Michael Wilde
>>>> Computation Institute, University of Chicago
>>>> Mathematics and Computer Science Division
>>>> Argonne National Laboratory
>>>>
>>>> _______________________________________________
>>>> Swift-devel mailing list
>>>> Swift-devel at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ketan
>>>
>>> --
>>> Michael Wilde
>>> Computation Institute, University of Chicago
>>> Mathematics and Computer Science Division
>>> Argonne National Laboratory
>>>
>>
>>
>>
>>
>> --
>> Ketan
>>
>>
>
>

-- 
Justin M Wozniak