[Swift-devel] CDM Tests

Mon Nov 14 14:05:00 CST 2011

Output files do not work correctly in CDM.  CDM creates symlinks to the files that are used in the DIRECT CDM directive. When the app outputs it overwrites the symlink in the work directory and *doesn't* update the original file.  So the check to see if the expected output is the same as the correct output will fail. How are you checking to see if the app correctly ran?  Is there an error or is the original file in the cwd empty?

----- Original Message -----
From: "Michael Wilde" <wilde at mcs.anl.gov>
To: "Ketan Maheshwari" <ketancmaheshwari at gmail.com>
Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
Sent: Monday, November 14, 2011 1:59:22 PM
Subject: Re: [Swift-devel] CDM Tests

I meant the command line to the app() program, not the commandline to the swift command.

- Mike

----- Original Message -----
> From: "Ketan Maheshwari" <ketancmaheshwari at gmail.com>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Monday, November 14, 2011 11:50:48 AM
> Subject: Re: [Swift-devel] CDM Tests
> All commandlines are in a run.sh script batched together. I am writing
> a README in the testsuite to document this.
> 
> 
> On Mon, Nov 14, 2011 at 1:46 PM, Michael Wilde < wilde at mcs.anl.gov >
> wrote:
> 
> 
> Ketan, in these tests, I thought that the names that the app actually
> receives on its command line is worth documenting. Does that show
> anything that the script writer should be aware of?
> 
> 
> - Mike
> 
> ----- Original Message -----
> 
> 
> 
> > From: "Michael Wilde" < wilde at mcs.anl.gov >
> > To: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
> > Cc: "Swift Devel" < swift-devel at ci.uchicago.edu >
> > Sent: Monday, November 14, 2011 11:39:10 AM
> > Subject: Re: [Swift-devel] CDM Tests
> > Even though the tests worked in each of these 4 cases, the question
> > remains of "did the app see the same path names" and "did copies get
> > done"?
> >
> > So I think you need to dig deeper and explain the difference between
> > what CDM does for the direct case, and how (if at all) the wrapper
> > mode affects things.
> >
> > Its equally possible that CDM overrode the wrapper mode arg. Just
> > specifying the wrapper mode arg will not, I think, achieve the
> > elimination of the "copy to shared workflow directory" that "direct
> > mode" accomplishes.
> >
> > - Mike
> >
> > ----- Original Message -----
> > > From: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
> > > To: "Michael Wilde" < wilde at mcs.anl.gov >
> > > Cc: "Swift Devel" < swift-devel at ci.uchicago.edu >, "Justin M
> > > Wozniak"
> > > < wozniak at mcs.anl.gov >
> > > Sent: Monday, November 14, 2011 11:29:36 AM
> > > Subject: Re: [Swift-devel] CDM Tests
> > > Mike,
> > >
> > >
> > > In my initial email I said:
> > >
> > >
> > > ... when specifying the relative option on the config for
> > > wrapper.invocation.mode property the script works regardless of
> > > the
> > > paths of input/output.
> > >
> > >
> > > Which made me conclude that the option specified in the config for
> > > wrapper.invocation.mode property overrides the CDM policies.
> > >
> > >
> > > On Mon, Nov 14, 2011 at 1:21 PM, Michael Wilde < wilde at mcs.anl.gov
> > > >
> > > wrote:
> > >
> > >
> > > Ketan,
> > >
> > > I thought your initial email stated that the 4 cases of relative
> > > path
> > > names "worked". But this latter comment indicates that there is
> > > some
> > > kind of problem even in the relative cases.
> > >
> > > Can you clarify?
> > >
> > >
> > > - Mike
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
> > >
> > >
> > >
> > > > To: "Justin M Wozniak" < wozniak at mcs.anl.gov >
> > > > Cc: "Michael Wilde" < wilde at mcs.anl.gov >, "Swift Devel" <
> > > > swift-devel at ci.uchicago.edu >
> > > > Sent: Monday, November 14, 2011 11:01:35 AM
> > > > Subject: Re: [Swift-devel] CDM Tests
> > > > Given the CDM's support for relative pathnames, I would expect
> > > > the
> > > > case of relative inputs, relative outputs would succeed
> > > > irrespective
> > > > of config. However, from the tests, it seems that the config
> > > > option
> > > > is
> > > > overriding CDM directives.
> > > >
> > > >
> > > > On Mon, Nov 14, 2011 at 12:44 PM, Justin M Wozniak <
> > > > wozniak at mcs.anl.gov > wrote:
> > > >
> > > >
> > > >
> > > > CDM DIRECT does not currently support absolute path names. I
> > > > think
> > > > we
> > > > could provide something simple like "" for the field, as you
> > > > suggest.
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, 14 Nov 2011, Michael Wilde wrote:
> > > >
> > > > > I would expect that for absolute path mappings, the user would
> > > > > specify
> > > > > either :: or "/" as the path in the CDM DIRECT pattern. In
> > > > > other
> > > > > words,
> > > > > the user is saying "I have mapped the object to an absolute
> > > > > path
> > > > > name,
> > > > > and I want the app to get exactly this name". Hence I think
> > > > > that
> > > > > the
> > > > > best "location" field for a DIRECT rule is null ("").
> > > > >
> > > > > Justin, can you clarify what the code expects for this case?
> > > > >
> > > > > - Mike
> > > > >
> > > > > ----- Original Message -----
> > > > >> From: "Jonathan Monette" < jonmon at mcs.anl.gov >
> > > > >> To: "Michael Wilde" < wilde at mcs.anl.gov >
> > > > >> Cc: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >, "Swift
> > > > >> Devel" < swift-devel at ci.uchicago.edu >
> > > > >> Sent: Monday, November 14, 2011 9:13:01 AM
> > > > >> Subject: Re: [Swift-devel] CDM Tests
> > > > >> I couldn't tell from the excel(using my phone) but what was
> > > > >> absolute?
> > > > >> If I am not mistaken what CDM does is takes the path that is
> > > > >> specified
> > > > >> in the CDM file(which should be absolute) and appends the
> > > > >> file
> > > > >> path
> > > > >> that is used in the swift script to it(which should be
> > > > >> relative).
> > > > >> So
> > > > >> if they are both absolute I would expect it to fail.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Nov 14, 2011, at 10:56 AM, Michael Wilde <
> > > > >> wilde at mcs.anl.gov
> > > > >> >
> > > > >> wrote:
> > > > >>
> > > > >>> Thanks, Ketan - very nice tests and summary document.
> > > > >>>
> > > > >>> I should point out for those that didn't open the doc: the 4
> > > > >>> tests
> > > > >>> with absolute pathnames are failing.
> > > > >>>
> > > > >>> Ketan, can you work with Justin to see if this is a bug, or
> > > > >>> if
> > > > >>> the
> > > > >>> CDM directive needs to be coded differently for absolute
> > > > >>> paths?
> > > > >>> Then
> > > > >>> please test a fix, and as we discussed adapt the tests with
> > > > >>> annotations to enhance the User Guide section on CDM.
> > > > >>>
> > > > >>> - Mike
> > > > >>>
> > > > >>>
> > > > >>> ----- Original Message -----
> > > > >>>> From: "Ketan Maheshwari" < ketancmaheshwari at gmail.com >
> > > > >>>> To: "Swift Devel" < swift-devel at ci.uchicago.edu >
> > > > >>>> Sent: Monday, November 14, 2011 8:13:48 AM
> > > > >>>> Subject: [Swift-devel] CDM Tests
> > > > >>>> Hello,
> > > > >>>>
> > > > >>>>
> > > > >>>> I had a discussion with Mike about testing the CDM behavior
> > > > >>>> for
> > > > >>>> the
> > > > >>>> following cases:
> > > > >>>> 1. full versus relative paths for input
> > > > >>>> 2. full versus relative paths for output
> > > > >>>> 3. relative versus absolute option in config property:
> > > > >>>> wrapper.invocation.mode
> > > > >>>>
> > > > >>>>
> > > > >>>> In this regard, I made 8 tests for all the above
> > > > >>>> combinations.
> > > > >>>> I
> > > > >>>> used
> > > > >>>> simple local provider in this first set of tests.
> > > > >>>>
> > > > >>>>
> > > > >>>> From the tests it seems that when specifying the relative
> > > > >>>> option
> > > > >>>> on
> > > > >>>> the config for wrapper.invocation.mode property the script
> > > > >>>> works
> > > > >>>> regardless of the paths of input/output.
> > > > >>>>
> > > > >>>>
> > > > >>>> A detailed result with stdout, and paths to logs can be
> > > > >>>> found
> > > > >>>> here:
> > > > >>>> https://docs.google.com/spreadsheet/ccc?key=0AmvYSwENKFY9dG44V2VjRXJlUmZLNG9saERFeWZDcUE
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> The tests are in my CI dir: /home/ketan/cdm_tests
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> Regards, --
> > > > >>>> Ketan
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> _______________________________________________
> > > > >>>> Swift-devel mailing list
> > > > >>>> Swift-devel at ci.uchicago.edu
> > > > >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > >>>
> > > > >>> --
> > > > >>> Michael Wilde
> > > > >>> Computation Institute, University of Chicago
> > > > >>> Mathematics and Computer Science Division
> > > > >>> Argonne National Laboratory
> > > > >>>
> > > > >>> _______________________________________________
> > > > >>> Swift-devel mailing list
> > > > >>> Swift-devel at ci.uchicago.edu
> > > > >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > >
> > > > >
> > > >
> > > > --
> > > > Justin M Wozniak
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Swift-devel mailing list
> > > > Swift-devel at ci.uchicago.edu
> > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Ketan
> > >
> > > --
> > > Michael Wilde
> > > Computation Institute, University of Chicago
> > > Mathematics and Computer Science Division
> > > Argonne National Laboratory
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Ketan
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> 
> 
> 
> 
> --
> Ketan

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory

_______________________________________________
Swift-devel mailing list
Swift-devel at ci.uchicago.edu
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel