[Swift-user] Custom/External Mappers

Fri Sep 14 13:58:58 CDT 2012

Robin, adding this line to your mapper to handle the unexpected -line argument makes the example you sent work now:

    -line)  ;;

I was unable to find where this change got into trunk, but I will ask on the swift-devel list (and in bug 829) to back off the change.

- Mike

----- Original Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "Robin Weiss" <robinweiss at uchicago.edu>
> Cc: swift-user at ci.uchicago.edu
> Sent: Friday, September 14, 2012 1:43:42 PM
> Subject: Re: [Swift-user] Custom/External Mappers
> OK, I'm revising my diagnosis here. It looks like a change was
> introduced into trunk which is adding an extra argument to the ext
> mapper invocation giving the line number in the .swift script from
> which it was called. This looks to me like some developer debugging
> that got committed by mistake. We will post to this list when its
> removed. You can see the problem by inserting a debug echo into your
> mapper:
> 
> in /autonfs/home/wilde/swift/lab/rwmapper.2012.0914/clusMapper_name.sh
> -name name_here -line 16
> 
> Your mapper (not surprisingly) rejects the -line 16 argument and
> fails.
> 
> That said, Swift needs to do a better job in handling and reporting
> failures in external mappers.
> 
> - Mike
> 
> ----- Original Message -----
> > From: "Michael Wilde" <wilde at mcs.anl.gov>
> > To: "Robin Weiss" <robinweiss at uchicago.edu>
> > Cc: swift-user at ci.uchicago.edu
> > Sent: Friday, September 14, 2012 1:32:44 PM
> > Subject: Re: [Swift-user] Custom/External Mappers
> > Ooops, my mistake, sorry.
> >
> > You are correct, the ext mapper args are passed as "-argname value".
> > The user guide is correct and that did not likely change in trunk.
> >
> > So the problem is elsewhere. I will investigate further.
> >
> > - Mike
> >
> >
> > ----- Original Message -----
> > > From: "Robin Weiss" <robinweiss at uchicago.edu>
> > > To: "Michael Wilde" <wilde at mcs.anl.gov>,
> > > swift-user at ci.uchicago.edu
> > > Sent: Friday, September 14, 2012 1:26:39 PM
> > > Subject: Re: [Swift-user] Custom/External Mappers
> > > Interesting. I wrote the custom mapper script expecting swift to
> > > call
> > > it
> > > with a whitespace (not equal sign) between the parameter switch
> > > and
> > > the
> > > value that goes with it (I.e.: ./clusMapper_name.sh -name
> > > name_goes_here).
> > > This is how the example in the swift users guide for 0.93
> > > (http://www.ci.uchicago.edu/swift/guides/release-0.93/userguide/userguide.h
> > > tml#_external_mapper) does it. The example from the users guide
> > > will
> > > also
> > > chokes if you put an equal sign instead of whitespace between the
> > > param
> > > name switch and the value.
> > >
> > > This issue popped up after we updated to the latest trunk version
> > > from
> > > 0.93. Did the way swift calls external mappers change (I.e. Equal
> > > sign
> > > separated instead of whitespace between param name and value)?
> > >
> > > Robin
> > >
> > >
> > > --
> > > Robin M. Weiss
> > > Research Programmer
> > > Research Computing Center
> > > The University of Chicago
> > > 6030 S. Ellis Ave., Suite 289C
> > > Chicago, IL 60637
> > > robinweiss at uchicago.edu <mailto:labello at uchicago.edu>
> > > 773.702.9030
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 9/14/12 1:14 PM, "Michael Wilde" <wilde at mcs.anl.gov> wrote:
> > >
> > > >Robin, I looked into this problem a bit. It seems that your
> > > >external
> > > >mapper scripts are not returning anything on standard output (and
> > > >an
> > > >error message on stderr) for the arguments that you are passing
> > > >in
> > > >the
> > > >failing cases.
> > > >
> > > >Here's what my tests show:
> > > >
> > > >$ ./clusMapper_name.sh
> > > >[0] out/.0.out
> > > >[1] out/.1.out
> > > >[2] out/.2.out
> > > >[3] out/.3.out
> > > >$
> > > >
> > > >$ ./clusMapper_name.sh -name=name_here
> > > >./clusMapper_name.sh: bad mapper args
> > > >$
> > > >
> > > >$ ./clusMapper_loop.sh -nfiles=3
> > > >./clusMapper_loop.sh: bad mapper args
> > > >$
> > > >
> > > >I dont see exactly why Swift is hanging in the cases where the
> > > >mappers
> > > >return nothing. I would have thought a nicer behavior would be to
> > > >(a)
> > > >exit complaining that the external mapper didnt return with exit
> > > >code
> > > >0,
> > > >or (b) proceed as if the arrays were not mapped, and use the
> > > >default
> > > >(concurrent) mapper.
> > > >
> > > >I suspect whats happening is that Swift is not closing the array
> > > >when
> > > >the
> > > >mapper fails, or some similar synchronization error. We'll need
> > > >to
> > > >look
> > > >into this. I've filed this as bugzilla bug # 829.
> > > >
> > > >- Mike
> > > >
> > > >----- Original Message -----
> > > >> From: "Robin Weiss" <robinweiss at uchicago.edu>
> > > >> To: swift-user at ci.uchicago.edu
> > > >> Sent: Monday, September 10, 2012 4:56:45 PM
> > > >> Subject: [Swift-user] Custom/External Mappers
> > > >> Howdy swifters,
> > > >>
> > > >>
> > > >> I am having some issues getting external mappers to work. In
> > > >> particular, swift appears to hang when you use a custom mapper
> > > >> that
> > > >> takes in one or more command line arguments.
> > > >>
> > > >>
> > > >> Attached is a tar ball with a basic example of the issue. You
> > > >> can
> > > >> uncomment each line in the mapper.swift script to see the
> > > >> behavior.
> > > >> I
> > > >> have included 4 versions of my external mapper, two work and
> > > >> two
> > > >> do
> > > >> not (see mapper.swift). Also included are the config, sites
> > > >> (using
> > > >> localhost), and tc files I'm using.
> > > >>
> > > >>
> > > >> I noticed this issue after moving from version 0.93 to trunk
> > > >> (Swift
> > > >> trunk swift-r5917 cog-r3463) when some scripts I had been using
> > > >> started to hang.
> > > >>
> > > >>
> > > >> Thanks in advance,
> > > >> Robin
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Robin M. Weiss
> > > >> Research Programmer
> > > >> Research Computing Center
> > > >> The University of Chicago
> > > >> 6030 S. Ellis Ave., Suite 289C
> > > >> Chicago, IL 60637
> > > >> robinweiss at uchicago.edu
> > > >> 773.702.9030
> > > >> _______________________________________________
> > > >> Swift-user mailing list
> > > >> Swift-user at ci.uchicago.edu
> > > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > > >
> > > >--
> > > >Michael Wilde
> > > >Computation Institute, University of Chicago
> > > >Mathematics and Computer Science Division
> > > >Argonne National Laboratory
> > > >
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory