<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div><span>Hi Justin,</span></div><div><br></div><div><span>Thank you for looking at the issue. <br></span></div><div><span><br></span></div><div><span>The benchmark was working when </span>JOB_THROTTLE = 0.05 at different scales (<span>nodes = 64, 128, 256</span>)<span>. But it didn't make any progress at high rates for a long time.</span></div><div><span><br></span></div><div><span>I think this is due to the storage </span><span>slowdown </span><span>(I was using GPFS to have both the worker directory and also to stage-out the files). <br></span></div><div><span>Now I changed my setup to stage-out to PVFS and <span style="font-weight: bold;">now the benchmark successfully works</span> with different scale (nodes = 64, 128, 256) at high job throttle rate (</span>JOB_THROTTLE = 1000<span>)
.</span></div><div><br><span></span></div><div><span>Thank you again.</span></div><div><br><span></span></div><div><span>Regards</span></div><div><span>Emalayan</span></div><div><br></div> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div dir="ltr"> <font face="Arial" size="2"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Justin M Wozniak <wozniak@mcs.anl.gov><br> <b><span style="font-weight: bold;">To:</span></b> Emalayan Vairavanathan <svemalayan@yahoo.com> <br><b><span style="font-weight: bold;">Cc:</span></b> "swift-devel@ci.uchicago.edu" <swift-devel@ci.uchicago.edu> <br> <b><span style="font-weight: bold;">Sent:</span></b> Tuesday, 10 April 2012 12:22 PM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [Swift-devel] Swift did not make progress with high throtteling
rate<br> </font> </div> <br>Hi Emalayan<br> Are you saying that this case does run with the default throttles and fails with jobThrottle=1000?<br> I just took a look at the log file. It looks like the jobs do get scheduled. Are there any -info files to look at?<br> Justin<br><br>On Tue, 10 Apr 2012, Emalayan Vairavanathan wrote:<br><br>> Hi All,<br>> <br>> I tired to run my pipeline-swift benchmark on GPFS+PVFS with 128 compute nodes (Surveyor), JOB_THROTTLE = 1000 and JOBS_PER_NODE = 4.<br>> <br>> I used GPFS as the central storage and PVFS as the intermediate storage. The benchmark did not make any progress and I found the following messages in the log file. (This happened even with MosaStore)<br>> <br>> <br>> 2012-04-10 18:18:36,710+0000 WARN HangChecke No events in 10s.<br>> 2012-04-10 18:18:36,717+0000 WARN HangChecker<br>> Registered
futures:<br>> file stage_2_output - F/stage_2_output[95]:file - Open<br>> file stage_1_output - F/stage_1_output[85]:file - Open<br>> file stage_3_output - F/stage_3_output[62]:file - Open<br>> file stage_3_output - F/stage_3_output[44]:file - Open<br>> file stage_1_output - F/stage_1_output[4]:file - Open<br>> file stage_2_output - F/stage_2_output[3]:file - Open<br>> file input_data - F/input_data[121]:file - Open<br>> file stage_1_output - F/stage_1_output[113]:file - Open<br>> file stage_1_output - F/stage_1_output[98]:file - Open<br>> <br>> I am using the swift version that I took from Justin's home directory 3 weeks before. <br>> <br>> Do you have any idea ? Does swift has problem with high throttling rate / jobs-per-node ? I have attached swift log file and the benchmark with this mail. I highly appreciate your suggestions.<br>> <br>> <br>> Thank you<br>> Emalayan<br><br>-- Justin M
Wozniak<br><br> </div> </div> </div></body></html>