[Swift-devel] Excessive object-closing messages in log

[Michael Wilde]

Ben, for a small swift script that iterates over a parameter array, I 
seem to be getting about (N^2)/2 log records regarding object closing.

The messages are of the form:

2008-03-23 18:30:36,877-0600 INFO  CloseDataset 
org.griphyn.vdl.mapping.RootDataNode identifier 
tag:benc at ci.uchicago.edu,2008:swift:dataset:20080323-1828-x14dlldb:720000003246 
with no value at dataset=ofile (closed)

For 500 entries, I had about 130K object closing log records; for 1000 
entries, over 500K:

bb$ grep -i -c closedataset run{341,342}/amps*.log
run341/amps1-20080323-1828-25xcue89.log:130259
run342/amps1-20080323-1935-su38n0k5.log:510509

341 is 500 jobs, 342 is 1000 jobs.

Is this log mechanism supposed to do that? If so, is that practical, as 
we want to test this script in the range of 1M jobs.

run342 is in swift-logs/wilde

The script is below.

My properties are:

sitedir.keep=true
lazy.errors=true
execution.retries=0
#kickstart.always.transfer=true

throttle.submit=off
throttle.host.submit=off
throttle.transfers=20
throttle.file.operations=20
throttle.score.job.factor=1000000
sitedir.keep=true


- Mike



type amout;

(amout ofile ) runam3 (string id , string dieselLowSLightLLProd, string 
dieselMedSLightLLProd)
{
   app { runam3 id dieselLowSLightLLProd dieselMedSLightLLProd ; }
}

type params {
   string id;
   string dieselLowSLightLLProd;
   string dieselMedSLightLLProd;
};

doall(params p[])
{
   foreach pset in p {
     amout ofile <single_file_mapper; file=@strcat("amdi.",pset.id)>;
     ofile = runam3(pset.id, pset.dieselLowSLightLLProd, 
pset.dieselMedSLightLLProd);
   }
}

// Main

params p[];
p = readdata("paramlist");
doall(p);