[Swift-devel] Issues with Montage & Swift-CDM
Emalayan Vairavanathan
svemalayan at yahoo.com
Fri Mar 23 05:00:14 CDT 2012
Hi Jon,
Please find my reply below.
[Jon]: I think until we have more information about Swift/CDM interaction(the
readData interaction Justin gave) you should have some files DEFAULT
where they will be stored on gpfs. In the SwiftMontage case those files are small and only used once or
twice at most.
[Emalayan] I believe Justin's reply would clear my doubts too.
Meanwhile I can proceed and try to integrate Montage on BG/P with MosaStore (by keeping some intermediate files on GPFS ). As an immediate next step first I will try the setup with GPFS as the intermediate storage (before using MosaStore).
Thank you
Emalayan
________________________________
From: Jonathan Monette <jonmon at mcs.anl.gov>
To: Emalayan Vairavanathan <svemalayan at yahoo.com>
Cc: Justin M Wozniak <wozniak at mcs.anl.gov>; matei <matei at ece.ubc.ca>; "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com>
Sent: Thursday, 22 March 2012 9:01 PM
Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
So Justin can elaborate more(probably) on the readData issue and CDM. This is actually the first I heard of that . I was under the same assumptions you were on CDM and Swift but Justin is the one who wrote CDM for Swift so he knows more of the pitfalls.
As for the action plan, I think until we have more information about Swift/CDM interaction(the readData interaction Justin gave) you should have some files DEFAULT where they will be stored on gpfs. In the SwiftMontage case those files are small and only used once or twice at most, in fact I believe the apps that generate those files run on localhost(at least that is how I tested them). You should not witness any fs performance difference but since the overall goal of mosa is to use it for all intermediate data we will probably need to modify the scripts. The only question is what to modify them too and for that we needs Justin's input. You can certainly proceed in setting up SwiftMontage for testing and start writing code to capture the performance numbers you need if you let some files DEFAULT, so this may not be a blocker at the moment.
Notice, that you do not need to specify a file to DEFAULT in CDM. If there is not a rule in the cdm rule file, it automatically defaults.
On Mar 22, 2012, at 6:28 PM, Emalayan Vairavanathan wrote:
Hi Justin and Jon,
>
>
>I thought the goal of having CDM is to provide a translation between the
file data type (used in swift) and the actual location of the files. This will help to avoid the actual location of the file being hard coded in the swift-script and also help swift to harness the platform specific data transfer mechanisms.
>
>
>But from what Justin said, it seems the issue is with the rules (usage of CDM_DIRECT Vs CDM DEFAULT). I did not understand how such translation layer get confused by CDM_DIRECT / CDM DEFAULT. Does readData() calles does not go through this translation layer ?
>
>
>
>May be I am wrong here. If so please correct me and provide more high level information.
>
>
>
>
>
>Jon: What is the action plan ? Do we need modification in the monage swift scripts ? or Do you suggest me to use CDM rules with CDM_DEFAULT for some files ? (In this case these intermediate files will be stored in GPFS)
>
>
>Thank you very much
>
>Emalayan
>
>
>
>
>________________________________
> From: Jonathan Monette <jonmon at mcs.anl.gov>
>To: Justin M Wozniak <wozniak at mcs.anl.gov>
>Cc: Emalayan Vairavanathan <svemalayan at yahoo.com>; matei <matei at ece.ubc.ca>; "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com>
>Sent: Thursday, 22 March 2012 3:11 PM
>Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
>
>So those files are in _concurrent. Those files(the ones mapped with the concurrent mapper) are read by readData or readData2. So if he uses fs_1.data from a previous email it works, which he said he confirmed.
>
>On Mar 22, 2012, at 5:03 PM, Justin M Wozniak wrote:
>
>>
>> I think SwiftMontage is trying to do a readData() on a CDM DIRECT file. This file is not accessible to the Swift engine, as it will be on the compute nodes in MosaStore.
>>
>> We need to enumerate the file names that must be accessible to Swift for readData(). These will have to be CDM DEFAULT. The rest can be CDM DIRECT.
>>
>> Justin
>>
>> On Thu, 22 Mar 2012, Justin M Wozniak wrote:
>>
>>> Ok, I can run the whole workflow with fs.data:
>>>
>>> rule .*raw_dir.* DIRECT /home/emalayan/app/montage-swift-cdm/SwiftMontage/scripts
>>> rule .*header.hdr.* DIRECT /home/emalayan/app/montage-swift-cdm/SwiftMontage/scripts
>>> # rule .*.* DIRECT /tmp/local
>>>
>>> I am now going to uncomment the last line...
>>>
>>> Emalayan, can you try to run that and see what happens?
>>>
>>> On Thu, 22 Mar 2012, Emalayan Vairavanathan wrote:
>>>
>>>> Thank you Jon. I am using swift-0.93 (this is in our Cluster). So I do not need to check for garbage-collector
errors.
>>>> I forgot to mention one point about concurrent mappers. The pipeline swift benchmark I wrote uses concurrent mapper too. Last week I ran the benchmark successfully regardless of the concurrent mappers location (I tried various location for concurrent mappers). So I guess the location of the concurrent mappers wont be an issue.
>>>> Thank you
>>>> Emalayan
>>>> ________________________________
>>>> From: Jonathan Monette <jonmon at mcs.anl.gov>
>>>> To: Emalayan Vairavanathan <svemalayan at yahoo.com> Cc: Justin M Wozniak <wozniak at mcs.anl.gov>; matei <matei at ece.ubc.ca>; "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com> Sent: Thursday, 22 March 2012 2:45 PM
>>>> Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
>>>> I'll run with those rules. I saw the garbage collection exception in the log file. The exception may be causing no harm, but it probably shouldn't be throwing an exception at all so this may help fix it in trunk.
>>>> But it just occurred to me, which swift are you using again? I am running with the copy I grabbed from Justin's directory. Are you
running that as well or with 0.93? If with 0.93 you will not see that exception because that version does not have the swift garbage collector so we should not waste time with that exception.
>>>> On Mar 22, 2012, at 4:40 PM, Emalayan Vairavanathan wrote:
>>>> Hi Jon,
>>>>> Thank you for you suggestions. By the way can you try to run motage with the CDM rules below and see whether the problem is with concurrent mapper and not because of location dependency ?
>>>>> rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts
>>>>> rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/
>>>>> rule .*final.* DIRECT
>>>> /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/
>>>>> rule .*.* DIRECT /tmp/local
>>>>> Meantime I will run the montage again and see whether swift
garbage-collector throws some error.
>>>>> Please let me know if you have a better idea to debug the problem.
>>>>> Thank you
>>>>> Emalayan
>>>>> ________________________________
>>>>> From: Jonathan Monette <jonmon at mcs.anl.gov>
>>>>> To: Emalayan Vairavanathan <svemalayan at yahoo.com> Cc: Justin M Wozniak <wozniak at mcs.anl.gov>; matei <matei at ece.ubc.ca>; "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com> Sent: Thursday, 22 March 2012 2:20 PM
>>>>> Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
>>>>> So I was able to run it on surveyor and it completed. I see those same SetFielfValue lines but I don't think that is an issue. That is just because I declare an array called projected_images and fill it up inside another function with files I used the regexp mapper on. However I do see the Swift garbage collector kicking in and throwing exceptions:
>>>>> 2012-03-22 20:58:03,070+0000 INFO FileGarbageCollector Failed to clean file://localhost/_concurrent/back_struct-d10301cf-5be0-4918-b0c2-49be758cf53a-7-array//elt-2.-field/b
>>>>>
java.lang.RuntimeException: org.globus.cog.abstraction.impl.file.FileNotFoundException: _concurrent/back_struct-d10301cf-5be0-4918-b0c2-49be758cf53a-7-array//elt-2.-field/b not found.
>>>>> at org.griphyn.vdl.mapping.AbsFile.clean(AbsFile.java:191)
>>>>> at org.griphyn.vdl.mapping.file.FileGarbageCollector.run(FileGarbageCollector.java:115)
>>>>> I am not sure if that may be causing problems for when Emalayan tries to run with CDM.
>>>>> Emalayan, it does not look like those scripts expect files to be in a certain location, at least that is not what is intended. In the main swiftscript, when you call the other functions you pass the directory name you want the intermediate files to be stored in. Then in the SwiftMontage_Batch functions it uses those directories you passed to map input/output files. The only things that are expected to be in certain places are the raw_dir and
header.hdr. Those have to be in the pwd. However I will continue debugging to make sure those assumptions I made are holding.
>>>>> As to the different cdm setups, I do use the concurrent mapper(files that get dumped to _concurrent) where swift decides on the names. I did this for a couple files that I did not care what they were named and they were small enough that I didn't care if they were staged in/out or not. CDM may not like that. Perhaps Swift expects those _concurrent files in a certain place but you told CDM to put them someplace different. I am not sure, that is just a hypothesis. I can always change the scripts to not use the concurrent mapper and use a better mapper for the CDM rules if that turns out to be the case.
>>>>> On Mar 22, 2012, at 4:02 PM, Emalayan Vairavanathan wrote:
>>>>> Hi Justin,
>>>>>> For me the setup in
/home/emalayan/app/montage-swift-cdm/SwiftMontage/scripts works.
>>>>>> I tried onlogin2.surveyor with the swift binaries located in /home/wozniak/Public/swift/bin/swift
>>>>>> May be you are using a different swift version or may be a different login machine. Thank you
>>>>>> Emalayan
>>>>>> ________________________________
>>>>>> From: Justin M Wozniak <wozniak at mcs.anl.gov>
>>>>>> To: Jonathan Monette <jonmon at mcs.anl.gov> Cc: Emalayan Vairavanathan <svemalayan at yahoo.com>; matei <matei at ece.ubc.ca>; "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com> Sent: Thursday, 22 March 2012 1:49 PM
>>>>>> Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
>>>>>> This run had no CDM
>>>> file.
>>>>>> On Thu, 22 Mar 2012, Jonathan Monette wrote:
>>>>>>> Was that with CDM in your run? I am going to take a look too as to why that is showing up.
>>>>>>> On Mar 22, 2012, at 3:42 PM, Justin M Wozniak wrote:
>>>>>>>> Ok, I can get it started but I get:
>>>>>>>> 2012-03-22
20:24:06,048+0000 INFO SetFieldValue Set: projected_images[0]=null
>>>>>>>> 2012-03-22 20:24:06,049+0000 INFO SetFieldValue Set: projected_images[6]=null
>>>>>>>> 2012-03-22 20:24:06,050+0000 INFO SetFieldValue Set: projected_images[7]=null
>>>>>>>> resulting in:
>>>>>>>> File not found: /gpfs/home/wozniak/SwiftMontage/scripts/./proj_dir/null
>>>>>>>> This looks like a Swift bug. However, do you guys have an existing workaround?
>>>>>>>> Thanks
>>>>>>>> On Thu, 22 Mar 2012, Emalayan
>>>> Vairavanathan wrote:
>>>>>>>>> Hi Justin,
>>>>>>>>> Please use ./run_local.sh to run the montage without cdm locally on the headnode.
>>>>>>>>> The rest of the scripts (run-workers.sh,
run.sh, run-swift.sh, main.sh) are written to run experiments in our cluster and wont work in Surveyor.
>>>>>>>>> Please let me know if you have questions.
>>>>>>>>> Thank you
>>>>>>>>> Emalayan
>>>>>>>>> ________________________________
>>>>>>>>> From: Justin M Wozniak <wozniak at mcs.anl.gov>
>>>>>>>>> To: Emalayan Vairavanathan <svemalayan at yahoo.com> Cc: "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com>; matei <matei at ece.ubc.ca> Sent: Thursday, 22 March 2012 11:54 AM
>>>>>>>>> Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
>>>>>>>>> On Wed, 21 Mar 2012, Emalayan Vairavanathan wrote:
>>>>>>>>>> But I just setup everything on Surveyor and it works locally on the head node. You can find the setup here.
>>>>>>>>>> /home/emalayan/app/montage-swift-cdm/SwiftMontage/scripts
>>>>>>>>> What is the entry
>>>> point?
>>>>>>>>> Are we missing common.sh?
>>>>>>>>> -- Justin M
>>>>
Wozniak
>>>>>>>> --
>>>>>>>> Justin M Wozniak_______________________________________________
>>>>>>>> Swift-devel mailing list
>>>>>>>> Swift-devel at ci.uchicago.edu
>>>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>>> --
>>>>>> Justin M Wozniak
>>>
>>>
>>
>> --
>> Justin M Wozniak_______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20120323/95e1419a/attachment.html>
More information about the Swift-devel
mailing list