[Swift-devel] testing replication

[Ben Clifford]

I was motivated to build a test framework for replication, and through it 
I have recreated (in exciting repeatable fashion) what superficially 
appears to be a problem that Xi Li encountered the other day.

I hacked (in racy, unundented glory) provider-local to be sometimes slow 
(where sometimes means 2 minute delay on every execution except the 
first). This is provider-wonky, which is in 
https://svn.ci.uchicago.edu/svn/vdl2/provider-wonky.

Add a dependency for provider-wonky and build Swift.

Set provider="wonky" for execution in sites.xml, instead of local 
provider.

Attempt to run test 062-two-in-a-row. Observe that it runs (with the first 
job finishing pretty much instantly, and the second job taking a couple of 
minutes).

Activate replication with:

replication.enabled=true
replication.min.queue.time=15

Attempt to run 062-to-in-a-row again. Observe that the workflow does not 
finish successfully (and that, at least superficially, it looks like the 
same problem Xi had).

This is what I see on my console (it takes about 10 minutes to run in 
total):

Swift svn swift-r1990 (Swift modified locally) cog-r2023 (CoG modified 
locally)

RunID: 20080522-1339-sygzk6qc
Progress: 
echo started
Submitting wonky job 0
Wonky job in queue, job number 0
not sleeping - this is the first job, 0
Wonky job running now
Wonky job completed with exitCode 0
Wonky job status COMPLETED 0
echo completed
echo started
Submitting wonky job 1
Wonky job in queue, job number 1
Progress:  Selecting site:1 Finished successfully:1
Submitting wonky job 2
Wonky job in queue, job number 2
Progress:  Selecting site:1 Finished successfully:1
Progress:  Selecting site:1 Finished successfully:1
Progress:  Selecting site:1 Finished successfully:1
Wonky job running now
Cancel called on wonky job 2
Wonky job completed with exitCode 0
Wonky job status COMPLETED 1
Submitting wonky job 3
Wonky job in queue, job number 3
Wonky job running now
Progress:  Executing:1 Finished successfully:1
Progress:  Executing:1 Finished successfully:1
Wonky job running now
Wonky job completed with exitCode 0
Wonky job status COMPLETED 3
Submitting wonky job 4
Wonky job in queue, job number 4
Progress:  Selecting site:1 Finished successfully:1
Submitting wonky job 5
Wonky job in queue, job number 5
Progress:  Selecting site:1 Finished successfully:1
Progress:  Selecting site:1 Finished successfully:1
Progress:  Selecting site:1 Finished successfully:1
Wonky job running now
Cancel called on wonky job 5
Wonky job completed with exitCode 0
Wonky job status COMPLETED 4
Submitting wonky job 6
Wonky job in queue, job number 6
Submitting wonky job 7
Wonky job in queue, job number 7
Progress:  Selecting site:1 Finished successfully:1
Wonky job running now
Progress:  Selecting site:1 Finished successfully:1
Progress:  Selecting site:1 Finished successfully:1
Wonky job running now
Wonky job running now
Wonky job completed with exitCode 0
Wonky job status COMPLETED 6
Wonky job completed with exitCode 0
Wonky job status COMPLETED 7
echo failed
Execution failed:
        Submitting wonky job 8
Wonky job in queue, job number 8
Multiple mappings pointing to the same file 
(localhost:062-two-in-a-row-20080522-1339-sygzk6qc/shared/062-two-in-a-row.b.out) 
detected.
SWIFT RETURN CODE NON-ZERO


--