[Swift-user] Custom/External Mappers

[Michael Wilde]

OK, I'm revising my diagnosis here. It looks like a change was introduced into trunk  which is adding an extra argument to the ext mapper invocation giving the line number in the .swift script from which it was called.  This looks to me like some developer debugging that got committed by mistake.  We will post to this list when its removed.  You can see the problem by inserting a debug echo into your mapper:

in /autonfs/home/wilde/swift/lab/rwmapper.2012.0914/clusMapper_name.sh -name name_here -line 16

Your mapper (not surprisingly) rejects the -line 16 argument and fails.

That said, Swift needs to do a better job in handling and reporting failures in external mappers.

- Mike

----- Original Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "Robin Weiss" <robinweiss at uchicago.edu>
> Cc: swift-user at ci.uchicago.edu
> Sent: Friday, September 14, 2012 1:32:44 PM
> Subject: Re: [Swift-user] Custom/External Mappers
> Ooops, my mistake, sorry.
> 
> You are correct, the ext mapper args are passed as "-argname value".
> The user guide is correct and that did not likely change in trunk.
> 
> So the problem is elsewhere. I will investigate further.
> 
> - Mike
> 
> 
> ----- Original Message -----
> > From: "Robin Weiss" <robinweiss at uchicago.edu>
> > To: "Michael Wilde" <wilde at mcs.anl.gov>, swift-user at ci.uchicago.edu
> > Sent: Friday, September 14, 2012 1:26:39 PM
> > Subject: Re: [Swift-user] Custom/External Mappers
> > Interesting. I wrote the custom mapper script expecting swift to
> > call
> > it
> > with a whitespace (not equal sign) between the parameter switch and
> > the
> > value that goes with it (I.e.: ./clusMapper_name.sh -name
> > name_goes_here).
> > This is how the example in the swift users guide for 0.93
> > (http://www.ci.uchicago.edu/swift/guides/release-0.93/userguide/userguide.h
> > tml#_external_mapper) does it. The example from the users guide will
> > also
> > chokes if you put an equal sign instead of whitespace between the
> > param
> > name switch and the value.
> >
> > This issue popped up after we updated to the latest trunk version
> > from
> > 0.93. Did the way swift calls external mappers change (I.e. Equal
> > sign
> > separated instead of whitespace between param name and value)?
> >
> > Robin
> >
> >
> > --
> > Robin M. Weiss
> > Research Programmer
> > Research Computing Center
> > The University of Chicago
> > 6030 S. Ellis Ave., Suite 289C
> > Chicago, IL 60637
> > robinweiss at uchicago.edu <mailto:labello at uchicago.edu>
> > 773.702.9030
> >
> >
> >
> >
> >
> >
> > On 9/14/12 1:14 PM, "Michael Wilde" <wilde at mcs.anl.gov> wrote:
> >
> > >Robin, I looked into this problem a bit. It seems that your
> > >external
> > >mapper scripts are not returning anything on standard output (and
> > >an
> > >error message on stderr) for the arguments that you are passing in
> > >the
> > >failing cases.
> > >
> > >Here's what my tests show:
> > >
> > >$ ./clusMapper_name.sh
> > >[0] out/.0.out
> > >[1] out/.1.out
> > >[2] out/.2.out
> > >[3] out/.3.out
> > >$
> > >
> > >$ ./clusMapper_name.sh -name=name_here
> > >./clusMapper_name.sh: bad mapper args
> > >$
> > >
> > >$ ./clusMapper_loop.sh -nfiles=3
> > >./clusMapper_loop.sh: bad mapper args
> > >$
> > >
> > >I dont see exactly why Swift is hanging in the cases where the
> > >mappers
> > >return nothing. I would have thought a nicer behavior would be to
> > >(a)
> > >exit complaining that the external mapper didnt return with exit
> > >code
> > >0,
> > >or (b) proceed as if the arrays were not mapped, and use the
> > >default
> > >(concurrent) mapper.
> > >
> > >I suspect whats happening is that Swift is not closing the array
> > >when
> > >the
> > >mapper fails, or some similar synchronization error. We'll need to
> > >look
> > >into this. I've filed this as bugzilla bug # 829.
> > >
> > >- Mike
> > >
> > >----- Original Message -----
> > >> From: "Robin Weiss" <robinweiss at uchicago.edu>
> > >> To: swift-user at ci.uchicago.edu
> > >> Sent: Monday, September 10, 2012 4:56:45 PM
> > >> Subject: [Swift-user] Custom/External Mappers
> > >> Howdy swifters,
> > >>
> > >>
> > >> I am having some issues getting external mappers to work. In
> > >> particular, swift appears to hang when you use a custom mapper
> > >> that
> > >> takes in one or more command line arguments.
> > >>
> > >>
> > >> Attached is a tar ball with a basic example of the issue. You can
> > >> uncomment each line in the mapper.swift script to see the
> > >> behavior.
> > >> I
> > >> have included 4 versions of my external mapper, two work and two
> > >> do
> > >> not (see mapper.swift). Also included are the config, sites
> > >> (using
> > >> localhost), and tc files I'm using.
> > >>
> > >>
> > >> I noticed this issue after moving from version 0.93 to trunk
> > >> (Swift
> > >> trunk swift-r5917 cog-r3463) when some scripts I had been using
> > >> started to hang.
> > >>
> > >>
> > >> Thanks in advance,
> > >> Robin
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >> Robin M. Weiss
> > >> Research Programmer
> > >> Research Computing Center
> > >> The University of Chicago
> > >> 6030 S. Ellis Ave., Suite 289C
> > >> Chicago, IL 60637
> > >> robinweiss at uchicago.edu
> > >> 773.702.9030
> > >> _______________________________________________
> > >> Swift-user mailing list
> > >> Swift-user at ci.uchicago.edu
> > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > >
> > >--
> > >Michael Wilde
> > >Computation Institute, University of Chicago
> > >Mathematics and Computer Science Division
> > >Argonne National Laboratory
> > >
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory