[Swift-devel] Performance problem with CDM direct processing

[Jonathan Monette]

Correct. I suspect if we can improve the performance of this section we can go from a run 12 hour run to a 6-8 hour run. 

The number of files that are being procesed by cdm look up is 320K.  What was observed was several processes were spawned for each file and took maybe a second to run(i think that was the time). 

Mike and me had a discussion on how we can replicate it with a simple test case to show the delay as well as some simple fixes to try out. 

----- Reply message -----
From: "Michael Wilde" <wilde at mcs.anl.gov>
Date: Mon, Aug 22, 2011 10:41 am
Subject: [Swift-devel] Performance problem with CDM direct processing
To: "Jonathan Monette" <jon.monette at gmail.com>, "Justin M Wozniak" <wozniak at mcs.anl.gov>
Cc: "swift-devel Devel" <swift-devel at ci.uchicago.edu>


Justin,

In testing Montage, Jon observed what looks like a performance bottleneck in the processing of CDM direct output passing.

I *think* what was happening was that a large number of jobs (say 25,000 or more, but I dont recall the exact number, it may have been larger) produced an output file, and all those files were being passed as input to a merge job.

What we observed was that the scripts being called from _swiftwrap (and perhaps some processing at the vdl-int.k level??? as well) were running very slowly, and that a fairly large number of scripts were being invoked per file. I think (but am not sure) that the high overhead was being observed at the start of the merge job in CDM scripts called by _swiftwrap.

Jon, can you explain what you know about this problem, and then lets see if we can enhance the performance?  This is now the main bottleneck in this application, which is otherwise now performing quite well.

Thanks,

- Mike
_______________________________________________
Swift-devel mailing list
Swift-devel at ci.uchicago.edu
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110822/a6da516e/attachment.html>