[Swift-user] Custom/External Mappers

Sat Sep 15 16:59:14 CDT 2012

This should be fixed in trunk now and thus -line wont be added any more.

- Mike

----- Original Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "Robin Weiss" <robinweiss at uchicago.edu>
> Cc: swift-user at ci.uchicago.edu
> Sent: Friday, September 14, 2012 1:58:58 PM
> Subject: Re: [Swift-user] Custom/External Mappers
> Robin, adding this line to your mapper to handle the unexpected -line
> argument makes the example you sent work now:
> 
> -line) ;;
> 
> I was unable to find where this change got into trunk, but I will ask
> on the swift-devel list (and in bug 829) to back off the change.
> 
> - Mike
> 
> 
> ----- Original Message -----
> > From: "Michael Wilde" <wilde at mcs.anl.gov>
> > To: "Robin Weiss" <robinweiss at uchicago.edu>
> > Cc: swift-user at ci.uchicago.edu
> > Sent: Friday, September 14, 2012 1:43:42 PM
> > Subject: Re: [Swift-user] Custom/External Mappers
> > OK, I'm revising my diagnosis here. It looks like a change was
> > introduced into trunk which is adding an extra argument to the ext
> > mapper invocation giving the line number in the .swift script from
> > which it was called. This looks to me like some developer debugging
> > that got committed by mistake. We will post to this list when its
> > removed. You can see the problem by inserting a debug echo into your
> > mapper:
> >
> > in
> > /autonfs/home/wilde/swift/lab/rwmapper.2012.0914/clusMapper_name.sh
> > -name name_here -line 16
> >
> > Your mapper (not surprisingly) rejects the -line 16 argument and
> > fails.
> >
> > That said, Swift needs to do a better job in handling and reporting
> > failures in external mappers.
> >
> > - Mike
> >
> > ----- Original Message -----
> > > From: "Michael Wilde" <wilde at mcs.anl.gov>
> > > To: "Robin Weiss" <robinweiss at uchicago.edu>
> > > Cc: swift-user at ci.uchicago.edu
> > > Sent: Friday, September 14, 2012 1:32:44 PM
> > > Subject: Re: [Swift-user] Custom/External Mappers
> > > Ooops, my mistake, sorry.
> > >
> > > You are correct, the ext mapper args are passed as "-argname
> > > value".
> > > The user guide is correct and that did not likely change in trunk.
> > >
> > > So the problem is elsewhere. I will investigate further.
> > >
> > > - Mike
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Robin Weiss" <robinweiss at uchicago.edu>
> > > > To: "Michael Wilde" <wilde at mcs.anl.gov>,
> > > > swift-user at ci.uchicago.edu
> > > > Sent: Friday, September 14, 2012 1:26:39 PM
> > > > Subject: Re: [Swift-user] Custom/External Mappers
> > > > Interesting. I wrote the custom mapper script expecting swift to
> > > > call
> > > > it
> > > > with a whitespace (not equal sign) between the parameter switch
> > > > and
> > > > the
> > > > value that goes with it (I.e.: ./clusMapper_name.sh -name
> > > > name_goes_here).
> > > > This is how the example in the swift users guide for 0.93
> > > > (http://www.ci.uchicago.edu/swift/guides/release-0.93/userguide/userguide.h
> > > > tml#_external_mapper) does it. The example from the users guide
> > > > will
> > > > also
> > > > chokes if you put an equal sign instead of whitespace between
> > > > the
> > > > param
> > > > name switch and the value.
> > > >
> > > > This issue popped up after we updated to the latest trunk
> > > > version
> > > > from
> > > > 0.93. Did the way swift calls external mappers change (I.e.
> > > > Equal
> > > > sign
> > > > separated instead of whitespace between param name and value)?
> > > >
> > > > Robin
> > > >
> > > >
> > > > --
> > > > Robin M. Weiss
> > > > Research Programmer
> > > > Research Computing Center
> > > > The University of Chicago
> > > > 6030 S. Ellis Ave., Suite 289C
> > > > Chicago, IL 60637
> > > > robinweiss at uchicago.edu <mailto:labello at uchicago.edu>
> > > > 773.702.9030
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 9/14/12 1:14 PM, "Michael Wilde" <wilde at mcs.anl.gov> wrote:
> > > >
> > > > >Robin, I looked into this problem a bit. It seems that your
> > > > >external
> > > > >mapper scripts are not returning anything on standard output
> > > > >(and
> > > > >an
> > > > >error message on stderr) for the arguments that you are passing
> > > > >in
> > > > >the
> > > > >failing cases.
> > > > >
> > > > >Here's what my tests show:
> > > > >
> > > > >$ ./clusMapper_name.sh
> > > > >[0] out/.0.out
> > > > >[1] out/.1.out
> > > > >[2] out/.2.out
> > > > >[3] out/.3.out
> > > > >$
> > > > >
> > > > >$ ./clusMapper_name.sh -name=name_here
> > > > >./clusMapper_name.sh: bad mapper args
> > > > >$
> > > > >
> > > > >$ ./clusMapper_loop.sh -nfiles=3
> > > > >./clusMapper_loop.sh: bad mapper args
> > > > >$
> > > > >
> > > > >I dont see exactly why Swift is hanging in the cases where the
> > > > >mappers
> > > > >return nothing. I would have thought a nicer behavior would be
> > > > >to
> > > > >(a)
> > > > >exit complaining that the external mapper didnt return with
> > > > >exit
> > > > >code
> > > > >0,
> > > > >or (b) proceed as if the arrays were not mapped, and use the
> > > > >default
> > > > >(concurrent) mapper.
> > > > >
> > > > >I suspect whats happening is that Swift is not closing the
> > > > >array
> > > > >when
> > > > >the
> > > > >mapper fails, or some similar synchronization error. We'll need
> > > > >to
> > > > >look
> > > > >into this. I've filed this as bugzilla bug # 829.
> > > > >
> > > > >- Mike
> > > > >
> > > > >----- Original Message -----
> > > > >> From: "Robin Weiss" <robinweiss at uchicago.edu>
> > > > >> To: swift-user at ci.uchicago.edu
> > > > >> Sent: Monday, September 10, 2012 4:56:45 PM
> > > > >> Subject: [Swift-user] Custom/External Mappers
> > > > >> Howdy swifters,
> > > > >>
> > > > >>
> > > > >> I am having some issues getting external mappers to work. In
> > > > >> particular, swift appears to hang when you use a custom
> > > > >> mapper
> > > > >> that
> > > > >> takes in one or more command line arguments.
> > > > >>
> > > > >>
> > > > >> Attached is a tar ball with a basic example of the issue. You
> > > > >> can
> > > > >> uncomment each line in the mapper.swift script to see the
> > > > >> behavior.
> > > > >> I
> > > > >> have included 4 versions of my external mapper, two work and
> > > > >> two
> > > > >> do
> > > > >> not (see mapper.swift). Also included are the config, sites
> > > > >> (using
> > > > >> localhost), and tc files I'm using.
> > > > >>
> > > > >>
> > > > >> I noticed this issue after moving from version 0.93 to trunk
> > > > >> (Swift
> > > > >> trunk swift-r5917 cog-r3463) when some scripts I had been
> > > > >> using
> > > > >> started to hang.
> > > > >>
> > > > >>
> > > > >> Thanks in advance,
> > > > >> Robin
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >>
> > > > >> Robin M. Weiss
> > > > >> Research Programmer
> > > > >> Research Computing Center
> > > > >> The University of Chicago
> > > > >> 6030 S. Ellis Ave., Suite 289C
> > > > >> Chicago, IL 60637
> > > > >> robinweiss at uchicago.edu
> > > > >> 773.702.9030
> > > > >> _______________________________________________
> > > > >> Swift-user mailing list
> > > > >> Swift-user at ci.uchicago.edu
> > > > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > > > >
> > > > >--
> > > > >Michael Wilde
> > > > >Computation Institute, University of Chicago
> > > > >Mathematics and Computer Science Division
> > > > >Argonne National Laboratory
> > > > >
> > >
> > > --
> > > Michael Wilde
> > > Computation Institute, University of Chicago
> > > Mathematics and Computer Science Division
> > > Argonne National Laboratory
> > >
> > > _______________________________________________
> > > Swift-user mailing list
> > > Swift-user at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory