[Swift-devel] CDM Tests

Jonathan Monette jonmon at mcs.anl.gov
Mon Nov 14 18:36:32 CST 2011 

The "do the right thing" is currently broken in CDM for output files.  A wrapper needs to be written currently around the app that updates the intended file and not the symlink in the work directory.  For instance, in the SwiftMontage scripts I wrap all the Montage calls in python scripts that checks to see if I am writing to a symlink, if I am writing to a symlink I write the what the symlink is pointing to instead of the symlink itself.  So unless a wrapper is written to handle the output files, CDM currently does not work properly.  This probably should be documented in the CDM section of the user guide as a caveat of using CDM for output files.

On Nov 14, 2011, at 2:47 PM, Michael Wilde wrote:

> I think the purpose of these tests was to find a default CDM DIRECT patter that would match all inout and output files and "do the right thing".
> 
> Tests that match specific pathnames are yet another (but different) useful case.
> 
> - Mike
> ----- Original Message -----
>> From: "Ketan Maheshwari" <ketancmaheshwari at gmail.com>
>> To: "Jon Monette" <jonmon at mcs.anl.gov>
>> Cc: "Michael Wilde" <wilde at mcs.anl.gov>, "Swift Devel" <swift-devel at ci.uchicago.edu>
>> Sent: Monday, November 14, 2011 12:43:13 PM
>> Subject: Re: [Swift-devel] CDM Tests
>> Yes, this is right. I added a rule for the outdir in the cdm file
>> (which was not added earlier) and all runs fails with the message:
>> file not found output/out?.txt.
>> 
>> 
>> On Mon, Nov 14, 2011 at 2:05 PM, Jon Monette < jonmon at mcs.anl.gov >
>> wrote:
>> 
>> 
>> Output files do not work correctly in CDM. CDM creates symlinks to the
>> files that are used in the DIRECT CDM directive. When the app outputs
>> it overwrites the symlink in the work directory and *doesn't* update
>> the original file. So the check to see if the expected output is the
>> same as the correct output will fail. How are you checking to see if
>> the app correctly ran? Is there an error or is the original file in
>> the cwd empty?
>> 
>> 
>> ----- Original Message -----
>> From: "Michael Wilde" < wilde at mcs.anl.gov >
>> To: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
>> Cc: "Swift Devel" < swift-devel at ci.uchicago.edu >
>> 
>> 
>> 
>> Sent: Monday, November 14, 2011 1:59:22 PM
>> Subject: Re: [Swift-devel] CDM Tests
>> 
>> I meant the command line to the app() program, not the commandline to
>> the swift command.
>> 
>> - Mike
>> 
>> ----- Original Message -----
>>> From: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
>>> To: "Michael Wilde" < wilde at mcs.anl.gov >
>>> Cc: "Swift Devel" < swift-devel at ci.uchicago.edu >
>>> Sent: Monday, November 14, 2011 11:50:48 AM
>>> Subject: Re: [Swift-devel] CDM Tests
>>> All commandlines are in a run.sh script batched together. I am
>>> writing
>>> a README in the testsuite to document this.
>>> 
>>> 
>>> On Mon, Nov 14, 2011 at 1:46 PM, Michael Wilde < wilde at mcs.anl.gov >
>>> wrote:
>>> 
>>> 
>>> Ketan, in these tests, I thought that the names that the app
>>> actually
>>> receives on its command line is worth documenting. Does that show
>>> anything that the script writer should be aware of?
>>> 
>>> 
>>> - Mike
>>> 
>>> ----- Original Message -----
>>> 
>>> 
>>> 
>>>> From: "Michael Wilde" < wilde at mcs.anl.gov >
>>>> To: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
>>>> Cc: "Swift Devel" < swift-devel at ci.uchicago.edu >
>>>> Sent: Monday, November 14, 2011 11:39:10 AM
>>>> Subject: Re: [Swift-devel] CDM Tests
>>>> Even though the tests worked in each of these 4 cases, the
>>>> question
>>>> remains of "did the app see the same path names" and "did copies
>>>> get
>>>> done"?
>>>> 
>>>> So I think you need to dig deeper and explain the difference
>>>> between
>>>> what CDM does for the direct case, and how (if at all) the wrapper
>>>> mode affects things.
>>>> 
>>>> Its equally possible that CDM overrode the wrapper mode arg. Just
>>>> specifying the wrapper mode arg will not, I think, achieve the
>>>> elimination of the "copy to shared workflow directory" that
>>>> "direct
>>>> mode" accomplishes.
>>>> 
>>>> - Mike
>>>> 
>>>> ----- Original Message -----
>>>>> From: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
>>>>> To: "Michael Wilde" < wilde at mcs.anl.gov >
>>>>> Cc: "Swift Devel" < swift-devel at ci.uchicago.edu >, "Justin M
>>>>> Wozniak"
>>>>> < wozniak at mcs.anl.gov >
>>>>> Sent: Monday, November 14, 2011 11:29:36 AM
>>>>> Subject: Re: [Swift-devel] CDM Tests
>>>>> Mike,
>>>>> 
>>>>> 
>>>>> In my initial email I said:
>>>>> 
>>>>> 
>>>>> ... when specifying the relative option on the config for
>>>>> wrapper.invocation.mode property the script works regardless of
>>>>> the
>>>>> paths of input/output.
>>>>> 
>>>>> 
>>>>> Which made me conclude that the option specified in the config
>>>>> for
>>>>> wrapper.invocation.mode property overrides the CDM policies.
>>>>> 
>>>>> 
>>>>> On Mon, Nov 14, 2011 at 1:21 PM, Michael Wilde <
>>>>> wilde at mcs.anl.gov
>>>>>> 
>>>>> wrote:
>>>>> 
>>>>> 
>>>>> Ketan,
>>>>> 
>>>>> I thought your initial email stated that the 4 cases of relative
>>>>> path
>>>>> names "worked". But this latter comment indicates that there is
>>>>> some
>>>>> kind of problem even in the relative cases.
>>>>> 
>>>>> Can you clarify?
>>>>> 
>>>>> 
>>>>> - Mike
>>>>> 
>>>>> 
>>>>> ----- Original Message -----
>>>>>> From: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
>>>>> 
>>>>> 
>>>>> 
>>>>>> To: "Justin M Wozniak" < wozniak at mcs.anl.gov >
>>>>>> Cc: "Michael Wilde" < wilde at mcs.anl.gov >, "Swift Devel" <
>>>>>> swift-devel at ci.uchicago.edu >
>>>>>> Sent: Monday, November 14, 2011 11:01:35 AM
>>>>>> Subject: Re: [Swift-devel] CDM Tests
>>>>>> Given the CDM's support for relative pathnames, I would expect
>>>>>> the
>>>>>> case of relative inputs, relative outputs would succeed
>>>>>> irrespective
>>>>>> of config. However, from the tests, it seems that the config
>>>>>> option
>>>>>> is
>>>>>> overriding CDM directives.
>>>>>> 
>>>>>> 
>>>>>> On Mon, Nov 14, 2011 at 12:44 PM, Justin M Wozniak <
>>>>>> wozniak at mcs.anl.gov > wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> CDM DIRECT does not currently support absolute path names. I
>>>>>> think
>>>>>> we
>>>>>> could provide something simple like "" for the field, as you
>>>>>> suggest.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Mon, 14 Nov 2011, Michael Wilde wrote:
>>>>>> 
>>>>>>> I would expect that for absolute path mappings, the user
>>>>>>> would
>>>>>>> specify
>>>>>>> either :: or "/" as the path in the CDM DIRECT pattern. In
>>>>>>> other
>>>>>>> words,
>>>>>>> the user is saying "I have mapped the object to an absolute
>>>>>>> path
>>>>>>> name,
>>>>>>> and I want the app to get exactly this name". Hence I think
>>>>>>> that
>>>>>>> the
>>>>>>> best "location" field for a DIRECT rule is null ("").
>>>>>>> 
>>>>>>> Justin, can you clarify what the code expects for this case?
>>>>>>> 
>>>>>>> - Mike
>>>>>>> 
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Jonathan Monette" < jonmon at mcs.anl.gov >
>>>>>>>> To: "Michael Wilde" < wilde at mcs.anl.gov >
>>>>>>>> Cc: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >,
>>>>>>>> "Swift
>>>>>>>> Devel" < swift-devel at ci.uchicago.edu >
>>>>>>>> Sent: Monday, November 14, 2011 9:13:01 AM
>>>>>>>> Subject: Re: [Swift-devel] CDM Tests
>>>>>>>> I couldn't tell from the excel(using my phone) but what was
>>>>>>>> absolute?
>>>>>>>> If I am not mistaken what CDM does is takes the path that
>>>>>>>> is
>>>>>>>> specified
>>>>>>>> in the CDM file(which should be absolute) and appends the
>>>>>>>> file
>>>>>>>> path
>>>>>>>> that is used in the swift script to it(which should be
>>>>>>>> relative).
>>>>>>>> So
>>>>>>>> if they are both absolute I would expect it to fail.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Nov 14, 2011, at 10:56 AM, Michael Wilde <
>>>>>>>> wilde at mcs.anl.gov
>>>>>>>>> 
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Thanks, Ketan - very nice tests and summary document.
>>>>>>>>> 
>>>>>>>>> I should point out for those that didn't open the doc: the
>>>>>>>>> 4
>>>>>>>>> tests
>>>>>>>>> with absolute pathnames are failing.
>>>>>>>>> 
>>>>>>>>> Ketan, can you work with Justin to see if this is a bug,
>>>>>>>>> or
>>>>>>>>> if
>>>>>>>>> the
>>>>>>>>> CDM directive needs to be coded differently for absolute
>>>>>>>>> paths?
>>>>>>>>> Then
>>>>>>>>> please test a fix, and as we discussed adapt the tests
>>>>>>>>> with
>>>>>>>>> annotations to enhance the User Guide section on CDM.
>>>>>>>>> 
>>>>>>>>> - Mike
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
>>>>>>>>>> To: "Swift Devel" < swift-devel at ci.uchicago.edu >
>>>>>>>>>> Sent: Monday, November 14, 2011 8:13:48 AM
>>>>>>>>>> Subject: [Swift-devel] CDM Tests
>>>>>>>>>> Hello,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> I had a discussion with Mike about testing the CDM
>>>>>>>>>> behavior
>>>>>>>>>> for
>>>>>>>>>> the
>>>>>>>>>> following cases:
>>>>>>>>>> 1. full versus relative paths for input
>>>>>>>>>> 2. full versus relative paths for output
>>>>>>>>>> 3. relative versus absolute option in config property:
>>>>>>>>>> wrapper.invocation.mode
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> In this regard, I made 8 tests for all the above
>>>>>>>>>> combinations.
>>>>>>>>>> I
>>>>>>>>>> used
>>>>>>>>>> simple local provider in this first set of tests.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> From the tests it seems that when specifying the relative
>>>>>>>>>> option
>>>>>>>>>> on
>>>>>>>>>> the config for wrapper.invocation.mode property the
>>>>>>>>>> script
>>>>>>>>>> works
>>>>>>>>>> regardless of the paths of input/output.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> A detailed result with stdout, and paths to logs can be
>>>>>>>>>> found
>>>>>>>>>> here:
>>>>>>>>>> https://docs.google.com/spreadsheet/ccc?key=0AmvYSwENKFY9dG44V2VjRXJlUmZLNG9saERFeWZDcUE
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> The tests are in my CI dir: /home/ketan/cdm_tests
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Regards, --
>>>>>>>>>> Ketan
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Swift-devel mailing list
>>>>>>>>>> Swift-devel at ci.uchicago.edu
>>>>>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Michael Wilde
>>>>>>>>> Computation Institute, University of Chicago
>>>>>>>>> Mathematics and Computer Science Division
>>>>>>>>> Argonne National Laboratory
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> Swift-devel mailing list
>>>>>>>>> Swift-devel at ci.uchicago.edu
>>>>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Justin M Wozniak
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Swift-devel mailing list
>>>>>> Swift-devel at ci.uchicago.edu
>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Ketan
>>>>> 
>>>>> --
>>>>> Michael Wilde
>>>>> Computation Institute, University of Chicago
>>>>> Mathematics and Computer Science Division
>>>>> Argonne National Laboratory
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Ketan
>>>> 
>>>> --
>>>> Michael Wilde
>>>> Computation Institute, University of Chicago
>>>> Mathematics and Computer Science Division
>>>> Argonne National Laboratory
>>>> 
>>>> _______________________________________________
>>>> Swift-devel mailing list
>>>> Swift-devel at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>> 
>>> --
>>> Michael Wilde
>>> Computation Institute, University of Chicago
>>> Mathematics and Computer Science Division
>>> Argonne National Laboratory
>>> 
>>> 
>>> 
>>> 
>>> 
>>> --
>>> Ketan
>> 
>> --
>> Michael Wilde
>> Computation Institute, University of Chicago
>> Mathematics and Computer Science Division
>> Argonne National Laboratory
>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> 
>> 
>> 
>> 
>> --
>> Ketan
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>

More information about the Swift-devel mailing list