[Swift-devel] Coaster provider staging data xfer problem

[Michael Wilde]

I just re-ran what I thought was my failing test, and it ran OK, but failed strangely in the swift cleanup process.

Localhost coasters; provider staging.

This run is at the moment on communicado in /tmp/wilde/run01 (Im running in /tmp due to the nature of the IO)

The test is running 9999 cat jobs; one file in and one out per job.  The file sizes are on order of <500KB each (random sizes).

all 9999 files were produced, but then I got a lot of unlink messages and some strange exit code 11 error.

The messages are in swift.stdouterr

The script eas executed using ./run.sh; tc and sites file are in that run01 dir.

This is worth looking at but low prio I think. I think the script terminated cleanly on smaller runs (-n=5, -n=100). So perhaps provider staging gets confused or has sync/mutex problems related to cleanup that occur at larger volumes of file copies???

At any rate, this was *not* the error that I was referring to in the message below; in that test, staging died in the middle of a run.  I will also try to test between two hosts.

- Mike


----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:

> On Sat, 2010-10-02 at 17:51 -0600, wilde at mcs.anl.gov wrote:
> > ----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> > > 
> > > Ok. I'll look at that. Just to be clear, you are talking about
> > > gridftp=coaster rather than use.provider.staging, right?
> > 
> > No, I dont *think* so!
> 
> Ok.
> 
> > 
> > What I meant above was provider staging via the coaster execution
> > provider, which is the only coaster based data transport technique
> I
> > knew of.
> > 
> > I'll try to replicate my test and send it.
> 
> Ok. I tried 1024 jobs, 8 concurrent, 7MB files and I can't reproduce
> it,
> so it may not be straightforward.
> 
> > 
> > I didnt know there was such a thing as gridftp=coaster!
> > 
> > Would that be done by saying <filesystem provider="coaster"> ?
> 
> Yes.
> 
> > I didnt know you could say either of those. Can you explain what
> that
> > would do and how to say it? Is it a different data provider path
> than
> > provider staging, but which still uses coasters?  Independent of
> > coaster execution? (I might be way off base here, sorry!)
> 
> Yes, and yes. Though I recommend provider staging.
> 
> Mihael

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory