[Swift-devel] Swift did not make progress with high throtteling rate

[Emalayan Vairavanathan]

Hi All,

I tired to run my pipeline-swift benchmark on GPFS+PVFS with 128 compute nodes (Surveyor), JOB_THROTTLE = 1000 and JOBS_PER_NODE = 4. 

I used GPFS as the central storage and PVFS as the intermediate storage. The benchmark did not make any progress and I found the following messages in the log file. (This happened even with MosaStore)


2012-04-10 18:18:36,710+0000 WARN  HangChecke No events in 10s.
2012-04-10 18:18:36,717+0000 WARN  HangChecker
Registered futures:
file stage_2_output - F/stage_2_output[95]:file - Open
file stage_1_output - F/stage_1_output[85]:file - Open
file stage_3_output - F/stage_3_output[62]:file - Open
file stage_3_output - F/stage_3_output[44]:file - Open
file stage_1_output - F/stage_1_output[4]:file - Open
file stage_2_output - F/stage_2_output[3]:file - Open
file input_data - F/input_data[121]:file - Open
file stage_1_output - F/stage_1_output[113]:file - Open
file stage_1_output - F/stage_1_output[98]:file - Open

I am using the swift version that I took from Justin's home directory 3 weeks before. 


Do you have any idea ? Does swift has problem with high throttling rate / jobs-per-node ? I have attached swift log file and the benchmark with this mail. I highly appreciate your suggestions.


Thank you
Emalayan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20120410/0698ba44/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: attachment.tar.gz
Type: application/x-gzip
Size: 75954 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20120410/0698ba44/attachment.bin>