From skrieder at iit.edu  Wed Jun  6 17:07:03 2012
From: skrieder at iit.edu (Scott Krieder)
Date: Wed, 6 Jun 2012 17:07:03 -0500
Subject: [Swift-user] Throttling within Swift
Message-ID: <CABp7gVxesEi3iqEx7-0-vrBng7qptz=4YkAttAzbRhMJbT9vUQ@mail.gmail.com>

Quick question about throttling within Swift:

I have a swift script that launches 64 binary files in parallel within a
foreach loop.

I would like Swift to throttle so that there are only 2 binaries running at
any given time. Then I would like to change the throttle for 4, 8, and 16.

I found several different layers at which I can change the throttle. For
the above configuration should I change the throttle in my sites.xml or do
I need to update the swift.properties file?

Thank you,
Scott

Scott Krieder
DataSys Laboratory
Illinois Institute of Technology
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120606/0b76357e/attachment.html>

From wilde at mcs.anl.gov  Wed Jun  6 17:29:25 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 6 Jun 2012 17:29:25 -0500 (CDT)
Subject: [Swift-user] Throttling within Swift
In-Reply-To: <CABp7gVxesEi3iqEx7-0-vrBng7qptz=4YkAttAzbRhMJbT9vUQ@mail.gmail.com>
Message-ID: <2051884677.12653.1339021765498.JavaMail.root@zimbra.anl.gov>

Scott, its best to adjust the throttle using the sites file. Use these two xml elements in your <pool> element:

    <profile namespace="karajan" key="jobThrottle">.63</profile>
    <profile namespace="karajan" key="initialScore">10000</profile>

Always set initialScore to 10000.
Set the actual throttle by setting jobThrottle to (N-1)/100 to run N jobs at a time.

So for N=2,4,8,16 set jobThrottle to .01, .03, .07, and .15

Thats admittedly rather arcane, but works that way for historical reasons. We hope to simplify this in the future.

- Mike



----- Original Message -----
> From: "Scott Krieder" <skrieder at iit.edu>
> To: swift-user at ci.uchicago.edu
> Sent: Wednesday, June 6, 2012 5:07:03 PM
> Subject: [Swift-user] Throttling within Swift
> Quick question about throttling within Swift:
> 
> 
> I have a swift script that launches 64 binary files in parallel within
> a foreach loop.
> 
> 
> I would like Swift to throttle so that there are only 2 binaries
> running at any given time. Then I would like to change the throttle
> for 4, 8, and 16.
> 
> 
> I found several different layers at which I can change the throttle.
> For the above configuration should I change the throttle in my
> sites.xml or do I need to update the swift.properties file?
> 
> 
> Thank you,
> Scott
> 
> 
> Scott Krieder
> DataSys Laboratory
> Illinois Institute of Technology
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From lpesce at uchicago.edu  Fri Jun  8 14:39:19 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Fri, 8 Jun 2012 14:39:19 -0500
Subject: [Swift-user] ram disk with swift
Message-ID: <B976B06A-D7C4-4017-9CD4-B3C3F2493739@uchicago.edu>

Mike,
We were talking about putting the input files for Taka's calculations (the X.mat files), which are rather small ~ 60Kb or so, in ram disk and try to use them for all the apps (that use the same file). giving the size, we could even load up all of them for each node (7*3*60=1.2MB, that is zero) to reduce traffic towards /lustre and thus avoid the meta-data slow downs that Beagle seems to have.

If we copy files in /tmp (as far as I understand it would be ramdisk) on the each node through swift, would it be there for all the apps?

Thanks,

Lorenzo

From wilde at mcs.anl.gov  Fri Jun  8 14:53:25 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 8 Jun 2012 14:53:25 -0500 (CDT)
Subject: [Swift-user] ram disk with swift
In-Reply-To: <B976B06A-D7C4-4017-9CD4-B3C3F2493739@uchicago.edu>
Message-ID: <1328245645.3177.1339185205054.JavaMail.root@zimbra.anl.gov>


> We were talking about putting the input files for Taka's calculations
> (the X.mat files), which are rather small ~ 60Kb or so, in ram disk
> and try to use them for all the apps (that use the same file). giving
> the size, we could even load up all of them for each node
> (7*3*60=1.2MB, that is zero) to reduce traffic towards /lustre and
> thus avoid the meta-data slow downs that Beagle seems to have.
> 
> If we copy files in /tmp (as far as I understand it would be ramdisk)
> on the each node through swift, would it be there for all the apps?

One way to do this, which has worked well for us is this:

- in the app wrapper script, create a function that loads the constant input data

- in this function, using directory-based locking logic to:

  -- create a data dir under /tmp
  -- fill the dir from a tarball

- in the app wrapper logic, if the /tmp/data dir is there, use it
  else call the function to create it.

This logic uses the fact that directory creation is atomic, so you can use that to prevent races where two apps running on the same node would each try to create the dir.

David, can you post the logic you developed for the above?

Thanks,

- Mike


From hockyg at gmail.com  Fri Jun  8 15:01:54 2012
From: hockyg at gmail.com (Glen Hocky)
Date: Fri, 8 Jun 2012 16:01:54 -0400
Subject: [Swift-user] ram disk with swift
In-Reply-To: <1328245645.3177.1339185205054.JavaMail.root@zimbra.anl.gov>
References: <B976B06A-D7C4-4017-9CD4-B3C3F2493739@uchicago.edu>
	<1328245645.3177.1339185205054.JavaMail.root@zimbra.anl.gov>
Message-ID: <CAGMfakmY=_531AsCrfjE3oncTODW1Fd4qxQ6iE2oXm=Fv8jzxw@mail.gmail.com>

Mike, Lorenzo,
Probably David will have something that will work but in case this helps,
this is the (a) routine you and I developed together for BGP some years ago
(July 08 apparently). Before running a few variables have to be set up so I
also included the setupvars script we had for reference...

function loadfile
{
        cd $rootdir
        local ARCHIVE=$1
        local ARCHIVE_NAME=`basename $ARCHIVE`
        local START_LOCK=$ARCHIVE_NAME-oops-cache-lock-started
        local END_LOCK=$ARCHIVE_NAME-oops-cache-lock-finished
        if [ ! -d ${START_LOCK} ];then
                mkdir ${START_LOCK}
                EXITCODE=$?
                if [ "${EXITCODE}" -ne "0" ];then
                    echo "waiting for data to be copied"
                    while [ ! -d ${END_LOCK} ];do
                        sleep 1
                    done
                else
                        echo "Copying $ARCHIVE to $rootdir"
                        cp -v $ARCHIVE $rootdir
                        tar xzvf $ARCHIVE_NAME
                        rm $ARCHIVE_NAME
                        mkdir ${END_LOCK}
                fi
        else
                while [ ! -d ${END_LOCK} ];do
                        sleep 1
                done
        fi
        return 0
}

function setupvars
{
        export PATH=/fuse/bin:/fuse/usr/bin:$PATH
        rootdir=/dev/shm
        data_location=$1
        protein=$2
        round=$3
        jobid=$4
        killtime=$5

        loadseed=1

        usagestring="$0 DATA_LOCATION PROTEIN ROUND JOBID KILLTIME"
        if [ -z $5 ];then
                echo $usagestring
                echo "Example: $0 /home/hockyg/glOOPS/bgprt 1ubq 1 1 30"
                exit 27
        fi

        oops_pack=$data_location/rt-base.tar.gz
        rtbasedir=$rootdir/rt-base
        input_pack=$data_location/input-$protein.tar.gz
        inputdir=$rootdir/input-$protein
        starttime=`date +%s`
        protid=$protein-$round-$jobid
        protdir=$rootdir/$protid
        timeshift=$(( $jobid % 60 ))

        mkdir -p $protdir/output

        returndir=$data_location/output/$protein/$round
        outdir=$protdir/output
        runoopslog=$protdir/output/$protid.runoopslog
        oopsreturnfile=$rootdir/$protid-done

        if [ ! -e $returndir ];then
                mkdir -p $returndir
        fi

        if [ ! -z $loadseed ];then
                seedfile=$protdir/output/$protid.random
                let seed=`expr $round*10000+$jobid`
                #echo "Writing seed $seed to $seedfile"
                echo $seed > $seedfile
        fi
}



On Fri, Jun 8, 2012 at 3:53 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

>
> > We were talking about putting the input files for Taka's calculations
> > (the X.mat files), which are rather small ~ 60Kb or so, in ram disk
> > and try to use them for all the apps (that use the same file). giving
> > the size, we could even load up all of them for each node
> > (7*3*60=1.2MB, that is zero) to reduce traffic towards /lustre and
> > thus avoid the meta-data slow downs that Beagle seems to have.
> >
> > If we copy files in /tmp (as far as I understand it would be ramdisk)
> > on the each node through swift, would it be there for all the apps?
>
> One way to do this, which has worked well for us is this:
>
> - in the app wrapper script, create a function that loads the constant
> input data
>
> - in this function, using directory-based locking logic to:
>
>  -- create a data dir under /tmp
>  -- fill the dir from a tarball
>
> - in the app wrapper logic, if the /tmp/data dir is there, use it
>  else call the function to create it.
>
> This logic uses the fact that directory creation is atomic, so you can use
> that to prevent races where two apps running on the same node would each
> try to create the dir.
>
> David, can you post the logic you developed for the above?
>
> Thanks,
>
> - Mike
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120608/d25bc512/attachment.html>

From davidk at ci.uchicago.edu  Fri Jun  8 16:34:43 2012
From: davidk at ci.uchicago.edu (David Kelly)
Date: Fri, 8 Jun 2012 16:34:43 -0500 (CDT)
Subject: [Swift-user] [Swift-devel] ram disk with swift
In-Reply-To: <B976B06A-D7C4-4017-9CD4-B3C3F2493739@uchicago.edu>
Message-ID: <1272810551.111958.1339191283314.JavaMail.root@zimbra-mb2.anl.gov>

Here is a portion of a script that that copies some frequently used data to a node's shared memory. I think it is pretty similar to the script that Glen sent earlier.

dssatdir=/dev/shm/dssat
dssattmp=/dev/shm/dssat.tmp

if [ ! -d $dssatdir ]; then
  if mkdir $dssattmp 2>/dev/null; then
    echo $$ copying
    savewd=$PWD
    cd $dssattmp

    mkdir -p refdata
    cp $refdata/* refdata/

    mkdir -p campaign
    cp $campaign/*.MZX campaign/

    mkdir -p binpath
    cp $binpath/*.EXE binpath/

    cd $savewd
    mv $dssattmp $dssatdir
  else
    while [ ! -d $dssatdir ]; do
     # echo $$ sleeping
      sleep 1;
    done
  fi
fi



----- Original Message -----
> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> To: "Michael Wilde" <wilde at mcs.anl.gov>, "Swift Devel" <swift-devel at ci.uchicago.edu>, swift-user at ci.uchicago.edu
> Cc: "Kazutaka Takahashi" <kazutaka at gmail.com>
> Sent: Friday, June 8, 2012 2:39:19 PM
> Subject: [Swift-devel] ram disk with swift
> Mike,
> We were talking about putting the input files for Taka's calculations
> (the X.mat files), which are rather small ~ 60Kb or so, in ram disk
> and try to use them for all the apps (that use the same file). giving
> the size, we could even load up all of them for each node
> (7*3*60=1.2MB, that is zero) to reduce traffic towards /lustre and
> thus avoid the meta-data slow downs that Beagle seems to have.
> 
> If we copy files in /tmp (as far as I understand it would be ramdisk)
> on the each node through swift, would it be there for all the apps?
> 
> Thanks,
> 
> Lorenzo
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel


From lpesce at uchicago.edu  Mon Jun 11 15:50:46 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Mon, 11 Jun 2012 15:50:46 -0500
Subject: [Swift-user] Throttling within Swift
In-Reply-To: <2051884677.12653.1339021765498.JavaMail.root@zimbra.anl.gov>
References: <2051884677.12653.1339021765498.JavaMail.root@zimbra.anl.gov>
Message-ID: <98B88641-711B-4F9A-B29F-0224FE61BDF8@uchicago.edu>

So, if I want to run 24 jobs per node on a 24 core node and use 100 nodes, I set it to 24?



On Jun 6, 2012, at 5:29 PM, Michael Wilde wrote:

> Scott, its best to adjust the throttle using the sites file. Use these two xml elements in your <pool> element:
> 
>    <profile namespace="karajan" key="jobThrottle">.63</profile>
>    <profile namespace="karajan" key="initialScore">10000</profile>
> 
> Always set initialScore to 10000.
> Set the actual throttle by setting jobThrottle to (N-1)/100 to run N jobs at a time.
> 
> So for N=2,4,8,16 set jobThrottle to .01, .03, .07, and .15
> 
> Thats admittedly rather arcane, but works that way for historical reasons. We hope to simplify this in the future.
> 
> - Mike
> 
> 
> 
> ----- Original Message -----
>> From: "Scott Krieder" <skrieder at iit.edu>
>> To: swift-user at ci.uchicago.edu
>> Sent: Wednesday, June 6, 2012 5:07:03 PM
>> Subject: [Swift-user] Throttling within Swift
>> Quick question about throttling within Swift:
>> 
>> 
>> I have a swift script that launches 64 binary files in parallel within
>> a foreach loop.
>> 
>> 
>> I would like Swift to throttle so that there are only 2 binaries
>> running at any given time. Then I would like to change the throttle
>> for 4, 8, and 16.
>> 
>> 
>> I found several different layers at which I can change the throttle.
>> For the above configuration should I change the throttle in my
>> sites.xml or do I need to update the swift.properties file?
>> 
>> 
>> Thank you,
>> Scott
>> 
>> 
>> Scott Krieder
>> DataSys Laboratory
>> Illinois Institute of Technology
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user



From wilde at mcs.anl.gov  Mon Jun 11 19:49:10 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Mon, 11 Jun 2012 19:49:10 -0500 (CDT)
Subject: [Swift-user] Throttling within Swift
In-Reply-To: <98B88641-711B-4F9A-B29F-0224FE61BDF8@uchicago.edu>
Message-ID: <294355157.1876.1339462150626.JavaMail.root@zimbra.anl.gov>


> From: "Lorenzo Pesce" <lpesce at uchicago.edu>

> So, if I want to run 24 jobs per node on a 24 core node and use 100
> nodes, I set it to 24?

Yes. 24 would enable 2400 jobs to get submitted to this site at once.

Thats what you would need if for example you set your sites.xml parameters for a coasters-enabled site to maxnodes 100 slote 1, or slots 100 maxnodes 1, or any similar factoring, with jobsPerNode 24.

There are three levels of throttling you need to be aware of (and can use): the two above, plus foreach.maxthreads, which limits each individual foreach.

- Mike

> 
> On Jun 6, 2012, at 5:29 PM, Michael Wilde wrote:
> 
> > Scott, its best to adjust the throttle using the sites file. Use
> > these two xml elements in your <pool> element:
> >
> >    <profile namespace="karajan" key="jobThrottle">.63</profile>
> >    <profile namespace="karajan" key="initialScore">10000</profile>
> >
> > Always set initialScore to 10000.
> > Set the actual throttle by setting jobThrottle to (N-1)/100 to run N
> > jobs at a time.
> >
> > So for N=2,4,8,16 set jobThrottle to .01, .03, .07, and .15
> >
> > Thats admittedly rather arcane, but works that way for historical
> > reasons. We hope to simplify this in the future.
> >
> > - Mike
> >
> >
> >
> > ----- Original Message -----
> >> From: "Scott Krieder" <skrieder at iit.edu>
> >> To: swift-user at ci.uchicago.edu
> >> Sent: Wednesday, June 6, 2012 5:07:03 PM
> >> Subject: [Swift-user] Throttling within Swift
> >> Quick question about throttling within Swift:
> >>
> >>
> >> I have a swift script that launches 64 binary files in parallel
> >> within
> >> a foreach loop.
> >>
> >>
> >> I would like Swift to throttle so that there are only 2 binaries
> >> running at any given time. Then I would like to change the throttle
> >> for 4, 8, and 16.
> >>
> >>
> >> I found several different layers at which I can change the
> >> throttle.
> >> For the above configuration should I change the throttle in my
> >> sites.xml or do I need to update the swift.properties file?
> >>
> >>
> >> Thank you,
> >> Scott
> >>
> >>
> >> Scott Krieder
> >> DataSys Laboratory
> >> Illinois Institute of Technology
> >> _______________________________________________
> >> Swift-user mailing list
> >> Swift-user at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From lpesce at uchicago.edu  Wed Jun 13 10:51:32 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Wed, 13 Jun 2012 10:51:32 -0500
Subject: [Swift-user] Question about swiftR temporary and work directories
Message-ID: <B98390B2-73B1-48DB-A598-EF45544D4BE5@uchicago.edu>

We are using swift R. Since we are working on a machine where the compute nodes see only the /lustre file system, we need to be able to set all the temporary directories there.
We know how to do that with swift, but swiftR doesn't seem to work that way.

This is the command:

> library(Swift)
> swiftInit(server="custom", tc.file="ROC.tc", cf.file="ROC.cf", sites.file="ROCSites.xml")
Working in /tmp/lpesce/SwiftR/swift.ZKHU
Running in /tmp/lpesce/SwiftR/swift.ZKHU (linked to /tmp/lpesce/SwiftR/swift.custom)
Started worker manager with pid 10153 

I would like both of those to point to /lustre/beagle/`whoami`/<some path>
where I can decide where to send them or at least that the place is under my /lustre folder.

Any ideas?

We set these variables before running R:
module load swift
export SWIFT_HEAP_MAX=1024M
export USER_HOME=/lustre/beagle/GCNet
export SWIFT_HOME=/soft/swift/0.93/

# set temporary directory references for R, currdir is /lustre/beagle/`whoami`
export TMP=$currdir/tmp
export TMPDIR=$TMP
export TEMP=$TMP
mkdir -p $TMP


This is the sites file:
 less ROCSites.xml
<config>
  <pool handle="pbs">
    <execution provider="coaster" jobmanager="local:pbs"/>
    <profile namespace="globus" key="project">CI-CCR000070</profile>
    

    <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>

    <profile namespace="globus" key="jobsPerNode">24</profile>
    <profile namespace="globus" key="maxTime">1800</profile>
    <profile namespace="globus" key="maxwalltime">0:2:00</profile>
    <profile namespace="globus" key="lowOverallocation">100</profile>
    <profile namespace="globus" key="highOverallocation">100</profile>

    <profile namespace="globus" key="slots">1</profile>
    <profile namespace="globus" key="nodeGranularity">1</profile>
    <profile namespace="globus" key="maxNodes">1</profile>

    <profile namespace="karajan" key="jobThrottle">.63</profile>
    <profile namespace="karajan" key="initialScore">10000</profile>

    <filesystem provider="local"/>
    <workdirectory>/lustre/beagle/lpesce/ROCSwift/swift.workdir</workdirectory>
  </pool>
</config>




From lpesce at uchicago.edu  Wed Jun 13 11:15:28 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Wed, 13 Jun 2012 11:15:28 -0500
Subject: [Swift-user] Question about swiftR temporary and work
	directories
In-Reply-To: <B98390B2-73B1-48DB-A598-EF45544D4BE5@uchicago.edu>
References: <B98390B2-73B1-48DB-A598-EF45544D4BE5@uchicago.edu>
Message-ID: <FD541CED-503D-4E80-925E-569D01BCAAB7@uchicago.edu>

I set user home wrong there, but it did not seem to be too relevant.

oddly, the tmp files seem to be set correctly

> tempdir()
[1] "/lustre/beagle/lpesce/ROCSwift/tmp/RtmpOzUNxM"
> tempfile()
[1] "/lustre/beagle/lpesce/ROCSwift/tmp/RtmpOzUNxM/file2a984c9ce26"




On Jun 13, 2012, at 10:51 AM, Lorenzo Pesce wrote:

> We are using swift R. Since we are working on a machine where the compute nodes see only the /lustre file system, we need to be able to set all the temporary directories there.
> We know how to do that with swift, but swiftR doesn't seem to work that way.
> 
> This is the command:
> 
>> library(Swift)
>> swiftInit(server="custom", tc.file="ROC.tc", cf.file="ROC.cf", sites.file="ROCSites.xml")
> Working in /tmp/lpesce/SwiftR/swift.ZKHU
> Running in /tmp/lpesce/SwiftR/swift.ZKHU (linked to /tmp/lpesce/SwiftR/swift.custom)
> Started worker manager with pid 10153 
> 
> I would like both of those to point to /lustre/beagle/`whoami`/<some path>
> where I can decide where to send them or at least that the place is under my /lustre folder.
> 
> Any ideas?
> 
> We set these variables before running R:
> module load swift
> export SWIFT_HEAP_MAX=1024M
> export USER_HOME=/lustre/beagle/GCNet
> export SWIFT_HOME=/soft/swift/0.93/
> 
> # set temporary directory references for R, currdir is /lustre/beagle/`whoami`
> export TMP=$currdir/tmp
> export TMPDIR=$TMP
> export TEMP=$TMP
> mkdir -p $TMP
> 
> 
> This is the sites file:
> less ROCSites.xml
> <config>
>  <pool handle="pbs">
>    <execution provider="coaster" jobmanager="local:pbs"/>
>    <profile namespace="globus" key="project">CI-CCR000070</profile>
> 
> 
>    <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
> 
>    <profile namespace="globus" key="jobsPerNode">24</profile>
>    <profile namespace="globus" key="maxTime">1800</profile>
>    <profile namespace="globus" key="maxwalltime">0:2:00</profile>
>    <profile namespace="globus" key="lowOverallocation">100</profile>
>    <profile namespace="globus" key="highOverallocation">100</profile>
> 
>    <profile namespace="globus" key="slots">1</profile>
>    <profile namespace="globus" key="nodeGranularity">1</profile>
>    <profile namespace="globus" key="maxNodes">1</profile>
> 
>    <profile namespace="karajan" key="jobThrottle">.63</profile>
>    <profile namespace="karajan" key="initialScore">10000</profile>
> 
>    <filesystem provider="local"/>
>    <workdirectory>/lustre/beagle/lpesce/ROCSwift/swift.workdir</workdirectory>
>  </pool>
> </config>
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user



From tim.g.armstrong at gmail.com  Wed Jun 13 11:46:10 2012
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Wed, 13 Jun 2012 11:46:10 -0500
Subject: [Swift-user] Question about swiftR temporary and work
	directories
In-Reply-To: <B98390B2-73B1-48DB-A598-EF45544D4BE5@uchicago.edu>
References: <B98390B2-73B1-48DB-A598-EF45544D4BE5@uchicago.edu>
Message-ID: <CAC0jiV4VdMGp2C_K+Lxy5UnO94PVTfmHLp7QpLBZz+UdFTLENw@mail.gmail.com>

There is an environment variable called SWIFTR_TMP that can control this.

Though... are you sure you want to do this?  It should work fine if it's
just pointing to /tmp/.  It isn't necessary that this temporary directory
be visible to nodes in the cluster, since it is just used for temporary
storage for the local processes. Many small files are created in the
directory, so I have no idea how it will work with lustre.

- Tim

On Wed, Jun 13, 2012 at 10:51 AM, Lorenzo Pesce <lpesce at uchicago.edu> wrote:

> We are using swift R. Since we are working on a machine where the compute
> nodes see only the /lustre file system, we need to be able to set all the
> temporary directories there.
> We know how to do that with swift, but swiftR doesn't seem to work that
> way.
>
> This is the command:
>
> > library(Swift)
> > swiftInit(server="custom", tc.file="ROC.tc", cf.file="ROC.cf",
> sites.file="ROCSites.xml")
> Working in /tmp/lpesce/SwiftR/swift.ZKHU
> Running in /tmp/lpesce/SwiftR/swift.ZKHU (linked to
> /tmp/lpesce/SwiftR/swift.custom)
> Started worker manager with pid 10153
>
> I would like both of those to point to /lustre/beagle/`whoami`/<some path>
> where I can decide where to send them or at least that the place is under
> my /lustre folder.
>
> Any ideas?
>
> We set these variables before running R:
> module load swift
> export SWIFT_HEAP_MAX=1024M
> export USER_HOME=/lustre/beagle/GCNet
> export SWIFT_HOME=/soft/swift/0.93/
>
> # set temporary directory references for R, currdir is
> /lustre/beagle/`whoami`
> export TMP=$currdir/tmp
> export TMPDIR=$TMP
> export TEMP=$TMP
> mkdir -p $TMP
>
>
> This is the sites file:
>  less ROCSites.xml
> <config>
>  <pool handle="pbs">
>    <execution provider="coaster" jobmanager="local:pbs"/>
>    <profile namespace="globus" key="project">CI-CCR000070</profile>
>
>
>    <profile namespace="globus"
> key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
>
>    <profile namespace="globus" key="jobsPerNode">24</profile>
>    <profile namespace="globus" key="maxTime">1800</profile>
>    <profile namespace="globus" key="maxwalltime">0:2:00</profile>
>    <profile namespace="globus" key="lowOverallocation">100</profile>
>    <profile namespace="globus" key="highOverallocation">100</profile>
>
>    <profile namespace="globus" key="slots">1</profile>
>    <profile namespace="globus" key="nodeGranularity">1</profile>
>    <profile namespace="globus" key="maxNodes">1</profile>
>
>    <profile namespace="karajan" key="jobThrottle">.63</profile>
>    <profile namespace="karajan" key="initialScore">10000</profile>
>
>    <filesystem provider="local"/>
>
>  <workdirectory>/lustre/beagle/lpesce/ROCSwift/swift.workdir</workdirectory>
>  </pool>
> </config>
>
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120613/355e65f4/attachment.html>

From lpesce at uchicago.edu  Wed Jun 13 12:12:21 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Wed, 13 Jun 2012 12:12:21 -0500
Subject: [Swift-user] Question about swiftR temporary and work
	directories
In-Reply-To: <CAC0jiV4VdMGp2C_K+Lxy5UnO94PVTfmHLp7QpLBZz+UdFTLENw@mail.gmail.com>
References: <B98390B2-73B1-48DB-A598-EF45544D4BE5@uchicago.edu>
	<CAC0jiV4VdMGp2C_K+Lxy5UnO94PVTfmHLp7QpLBZz+UdFTLENw@mail.gmail.com>
Message-ID: <26ADD60F-932B-4EE6-996A-0563A8C72422@uchicago.edu>

Yes, lustre will of course gel up and then freeze =)

You are right. I just wanted a way to change it in case I get a ramdisk explosion. =)

In fact  I am trying to use ramdisk with some other swift apps I am working on. 

Thanks a lot both for the answer and the advice.

On Jun 13, 2012, at 11:46 AM, Tim Armstrong wrote:

> There is an environment variable called SWIFTR_TMP that can control this.
> 
> Though... are you sure you want to do this?  It should work fine if it's just pointing to /tmp/.  It isn't necessary that this temporary directory be visible to nodes in the cluster, since it is just used for temporary storage for the local processes. Many small files are created in the directory, so I have no idea how it will work with lustre.
> 
> - Tim
> 
> On Wed, Jun 13, 2012 at 10:51 AM, Lorenzo Pesce <lpesce at uchicago.edu> wrote:
> We are using swift R. Since we are working on a machine where the compute nodes see only the /lustre file system, we need to be able to set all the temporary directories there.
> We know how to do that with swift, but swiftR doesn't seem to work that way.
> 
> This is the command:
> 
> > library(Swift)
> > swiftInit(server="custom", tc.file="ROC.tc", cf.file="ROC.cf", sites.file="ROCSites.xml")
> Working in /tmp/lpesce/SwiftR/swift.ZKHU
> Running in /tmp/lpesce/SwiftR/swift.ZKHU (linked to /tmp/lpesce/SwiftR/swift.custom)
> Started worker manager with pid 10153
> 
> I would like both of those to point to /lustre/beagle/`whoami`/<some path>
> where I can decide where to send them or at least that the place is under my /lustre folder.
> 
> Any ideas?
> 
> We set these variables before running R:
> module load swift
> export SWIFT_HEAP_MAX=1024M
> export USER_HOME=/lustre/beagle/GCNet
> export SWIFT_HOME=/soft/swift/0.93/
> 
> # set temporary directory references for R, currdir is /lustre/beagle/`whoami`
> export TMP=$currdir/tmp
> export TMPDIR=$TMP
> export TEMP=$TMP
> mkdir -p $TMP
> 
> 
> This is the sites file:
>  less ROCSites.xml
> <config>
>  <pool handle="pbs">
>    <execution provider="coaster" jobmanager="local:pbs"/>
>    <profile namespace="globus" key="project">CI-CCR000070</profile>
> 
> 
>    <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
> 
>    <profile namespace="globus" key="jobsPerNode">24</profile>
>    <profile namespace="globus" key="maxTime">1800</profile>
>    <profile namespace="globus" key="maxwalltime">0:2:00</profile>
>    <profile namespace="globus" key="lowOverallocation">100</profile>
>    <profile namespace="globus" key="highOverallocation">100</profile>
> 
>    <profile namespace="globus" key="slots">1</profile>
>    <profile namespace="globus" key="nodeGranularity">1</profile>
>    <profile namespace="globus" key="maxNodes">1</profile>
> 
>    <profile namespace="karajan" key="jobThrottle">.63</profile>
>    <profile namespace="karajan" key="initialScore">10000</profile>
> 
>    <filesystem provider="local"/>
>    <workdirectory>/lustre/beagle/lpesce/ROCSwift/swift.workdir</workdirectory>
>  </pool>
> </config>
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120613/cab74b5a/attachment.html>

From lpesce at uchicago.edu  Fri Jun 15 10:16:58 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Fri, 15 Jun 2012 10:16:58 -0500
Subject: [Swift-user] Issues with swiftR
Message-ID: <7EE13ECC-EB02-4917-9534-0865DD74B3D1@uchicago.edu>

Hi --

We need to run a set of simulations to test the statistical properties of an estimator.

This creates two sets of problems:

1) Our calculations involve exploring a number of parameter combinations, about 350 parameter combinations. For each combination we have 10,000 calculations or 3.5 million calculations. How am I going to use SwiftR for that? I can't apply to a 3.5 million element array and it makes little sense because the difference between each block of 10,000 is the random dataset, which is created following the same rules. Note that 10,000 calculations take about 160 hours on a single core, varying a little bit between dataset types.

2) Moreover, we need to generate random numbers, each of our calculations needs a randomly generate set of 50 to 200 numbers. We also need random numbers for the estimator. These random numbers ideally would be reproducible and not on disk. We need to use seeds, but the seeds need to be associated.

We can't split it into 350 instances of SwiftR of the cluster will die (probably). 


Suggestions?

From jmargolpeople at gmail.com  Fri Jun 15 12:49:29 2012
From: jmargolpeople at gmail.com (Jonathan Margoliash)
Date: Fri, 15 Jun 2012 12:49:29 -0500
Subject: [Swift-user] What does the mapper option padding do?
Message-ID: <CAJgtAspoG5rxewNOyTfrsy1i7CV8e0FgEP9-2Fbz7Z1QzpxqRg@mail.gmail.com>

Hello all,

I am new to the listhost and was wondering if this is the correct place to
ask questions about how Swift works. If so, my question is, what does the
mapper option padding do? I.e.

file inputs[] <simple_mapper; location=".", prefix="x", suffix=".mat",
padding=0>;

I couldn't find this discussed in the user guide on the Swift website.
Should I be looking for documentation elsewhere? Thanks for the help!

Jonathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120615/081b3b3d/attachment.html>

From ketancmaheshwari at gmail.com  Fri Jun 15 13:05:07 2012
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Fri, 15 Jun 2012 14:05:07 -0400
Subject: [Swift-user] What does the mapper option padding do?
In-Reply-To: <CAJgtAspoG5rxewNOyTfrsy1i7CV8e0FgEP9-2Fbz7Z1QzpxqRg@mail.gmail.com>
References: <CAJgtAspoG5rxewNOyTfrsy1i7CV8e0FgEP9-2Fbz7Z1QzpxqRg@mail.gmail.com>
Message-ID: <CAMUuviokvZPWOvHb40vjj7bJFyijnWiQ_AAAmckGzdsbavZ=JA@mail.gmail.com>

Padding basically puts that many number of digits after the prefix and
before suffix of the file. For instance, a padding of value 2 will result
in mappings as follows:

inputs[0] x00.mat
inputs[1] x01.mat
..
..
and so on.

On Fri, Jun 15, 2012 at 1:49 PM, Jonathan Margoliash <
jmargolpeople at gmail.com> wrote:

> Hello all,
>
> I am new to the listhost and was wondering if this is the correct place to
> ask questions about how Swift works. If so, my question is, what does the
> mapper option padding do? I.e.
>
> file inputs[] <simple_mapper; location=".", prefix="x", suffix=".mat",
> padding=0>;
>
> I couldn't find this discussed in the user guide on the Swift website.
> Should I be looking for documentation elsewhere? Thanks for the help!
>
> Jonathan
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>



-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120615/2d1a362d/attachment.html>

From jmargolpeople at gmail.com  Fri Jun 15 14:21:34 2012
From: jmargolpeople at gmail.com (Jonathan Margoliash)
Date: Fri, 15 Jun 2012 14:21:34 -0500
Subject: [Swift-user] Printing to the terminal and inadequate documentation
Message-ID: <CAJgtAsrioV4JX=fe=whK=d61_ssAuhu5UCHmYRkVw81d7ds4nA@mail.gmail.com>

Hello all,

Is it possible to print to the terminal from inside swift? I have not found
a print command inside swift, and the app

app sayHello () {
    echo "Hello!";
}

does not produce any output for me. Also, is there any better documentation
online than the user's manual?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120615/b82298b1/attachment.html>

From davidk at ci.uchicago.edu  Fri Jun 15 18:47:36 2012
From: davidk at ci.uchicago.edu (David Kelly)
Date: Fri, 15 Jun 2012 18:47:36 -0500 (CDT)
Subject: [Swift-user] Printing to the terminal and inadequate
 documentation
In-Reply-To: <CAJgtAsrioV4JX=fe=whK=d61_ssAuhu5UCHmYRkVw81d7ds4nA@mail.gmail.com>
Message-ID: <1511629120.30420.1339804056571.JavaMail.root@zimbra-mb2.anl.gov>

Hi Jonathan,

I think what you are looking for here is tracef(). It has a syntax similar to C's printf. 

tracef("Hello\n");

You don't need to call this from within an app function. More examples of how to use this are in the user guide at http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_tracef.

The best documentation that we have right now is the user guide and the tutorial at http://www.ci.uchicago.edu/swift/docs/index.php. If you notice that there is something missing or not clear, please let us know.

Thanks,
David

----- Original Message -----
> From: "Jonathan Margoliash" <jmargolpeople at gmail.com>
> To: swift-user at ci.uchicago.edu
> Sent: Friday, June 15, 2012 2:21:34 PM
> Subject: [Swift-user] Printing to the terminal and inadequate documentation
> Hello all,
> 
> 
> Is it possible to print to the terminal from inside swift? I have not
> found a print command inside swift, and the app
> 
> 
> app sayHello () {
> echo "Hello!";
> }
> 
> 
> does not produce any output for me. Also, is there any better
> documentation online than the user's manual?
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user


From iraicu at cs.iit.edu  Sat Jun 16 08:31:07 2012
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Sat, 16 Jun 2012 08:31:07 -0500
Subject: [Swift-user] CFP: The 13th IEEE/ACM Int. Symp. on Cluster,
 Cloud and Grid Computing (CCGrid) 2013
Message-ID: <4FDC8A9B.1000902@cs.iit.edu>

                       **** CALL FOR PAPERS ****

                 The 13th IEEE/ACM International Symposium on
                      Cluster, Cloud and Grid Computing
                             (CCGrid 2013)

           Delft University of Technology, Delft, the Netherlands

                            May 13-16, 2013

                http://www.pds.ewi.tudelft.nl/ccgrid2013



Rapid advances in architectures, networks, and systems and middleware technologies
are leading to new concepts in and platforms for computing, ranging from Clusters and
Grids to Clouds and Datacenters. CCGrid is a series of very successful conferences,
sponsored by the IEEE Computer Society Technical Committee on Scalable Computing (TCSC)
and the ACM, with the overarching goal of bringing together international researchers,
developers, and users to provide an international forum to present leading research
activities and results on a broad range of topics related to these concepts and platforms,
and their applications. The conference features keynotes, technical presentations,
workshops, tutorials, and posters, as well as the SCALE challenge featuring live demonstrations.

In 2013, CCGrid will come to the Netherlands for the first time, and will be held in Delft,
a historical, picturesque city that is less than one hour away from Amsterdam-Schiphol airport.
The main conference will be held on May 14-16 (Tuesday to Thursday), with tutorials and
affiliated workshops taking place on May 13 (Monday).


**** IMPORTANT DATES ****
Papers Due:                             12 November 2012
Author Notifications:                   24 January 2013
Final Papers Due:                       22 February 2013
  

**** TOPICS OF INTEREST ****
CCGrid 2013 will have a focus on important and immediate issues that are significantly
influencing all aspects of cluster, cloud and grid computing. Topics of interest include,
but are not limited to:

* Applications and Experiences: Applications to real and complex problems in science,
   engineering, business, and society; User studies; Experiences with large-scale deployments,
   systems, or applications
* Architecture: System architectures, design and deployment; Power and cooling; Security
   and reliability; High availability solutions
* Autonomic Computing and Cyberinfrastructure: Self-managed behavior, models and technologies;
   Autonomic paradigms and systems (control-based, bio-inspired, emergent, etc.); Bio-inspired
   optimizations and computing
* Cloud Computing: Cloud architectures; Software tools and techniques for clouds
* Multicore and Accelerator-based Computing: Software and application techniques to utilize
   multicore architectures and accelerators in clusters, grids, and clouds
* Performance Modeling and Evaluation: Performance prediction and modeling; Monitoring and
   evaluation tools; Analysis of system and application performance; Benchmarks and testbeds
* Programming Models, Systems, and Fault-Tolerant Computing: Programming models and
   environments for cluster, cloud, and grid computing; Fault-tolerant systems, programs and
   algorithms; Systems software to support efficient computing
* Scheduling and Resource Management: Techniques to schedule jobs and resources on cluster,
   cloud, and grid computing platforms; SLA definition and enforcement


**** PAPER SUBMISSION GUIDELINES ****
Authors are invited to submit papers electronically in PDF format. Submitted manuscripts
should be structured as technical papers and may not exceed 8 letter-size (8.5 x 11) pages
including figures, tables and references using the IEEE format for conference proceedings.
Submissions not conforming to these guidelines may be returned without review. Authors
should make sure that their file will print on a printer that uses letter-size (8.5 x 11)
paper. The official language of the conference is English. All manuscripts will be reviewed
and will be judged on correctness, originality, technical strength, significance, quality of
presentation, and interest and relevance to the conference attendees.

Submitted papers must represent original unpublished research that is not currently under
review for any other conference or journal. Papers not following these guidelines will be
rejected without review and further action may be taken, including (but not limited to)
notifications sent to the heads of the institutions of the authors and sponsors of the
conference. Submissions received after the due date, exceeding the page limit, or not
appropriately structured may not be considered. Authors may contact the conference chairs
for more information. The proceedings will be published through the IEEE Computer Society
Press, USA, and will be made available online through the IEEE Digital Library.


**** CALL FOR TUTORIAL AND WORKSHOP PROPOSALS ****
Tutorials and workshops affiliated with CCGrid 2013 will be held on May 13 (Monday). For
more information on the tutorials and workshops and for the complete Call for Tutorial and
Workshop Proposals, please see the conference website.


**** GENERAL CHAIR ****
Dick Epema, Delft University of Technology, the Netherlands


**** PROGRAM CHAIR ****
Thomas Fahringer, University of Innsbruck, Austria


**** PROGRAM VICE-CHAIRS ****
Rosa Badia, Barcelona Supercomputing Center, Spain
Henri Bal, Vrije Universiteit, the Netherlands
Marios Dikaiakos, University of Cyprus, Cyprus
Kirk Cameron, VirginiaTech, USA
Daniel Katz, University of Chicago & Argonne Nat Lab, USA
Kate Keahey, Argonne National Laboratory, USA
Martin Schulz, Lawrence Livermore National Laboratory, USA
Douglas Thain, University of Notre Dame, USA
Cheng-Zhong Xu, Shenzhen Inst. of Advanced Techn, China


**** WORKSHOPS CHAIR ****
Shantenu Jha, Rutgers and Louisana State University, USA


**** DOCTORAL SYMPOSIUM CHAIRS ****
Yogesh Simmhan, University of Southern California, USA
Ana Varbanescu, Delft Univ of Technology, the Netherlands


**** SUBMISSIONS AND PROCEEDINGS CHAIR ****
Pavan Balaji, Argonne National Laboratory, USA


**** FINANCE AND REGISTRATION CHAIR ****
Alexandru Iosup, Delft Univ of Technology, the Netherlands


**** PUBLICITY CHAIRS ****
Nazareno Andrade, University Federal de Campina Grance, Brazil
Gabriel Antoniu, INRIA, France
Bahman Javadi, University of Western Sysney, Australia
Ioan Raicu, Illinois Institute of Technology and Argonne National Laboratory, USA


**** CYBER CHAIR ****
Stephen van der Laan, Delft University of Technology, the Netherlands

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120616/1f1d6a87/attachment.html>

From wilde at mcs.anl.gov  Sun Jun 24 21:47:10 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 24 Jun 2012 21:47:10 -0500 (CDT)
Subject: [Swift-user] Question about swiftR temporary and work
 directories
In-Reply-To: <B98390B2-73B1-48DB-A598-EF45544D4BE5@uchicago.edu>
Message-ID: <2006547794.19762.1340592430721.JavaMail.root@zimbra.anl.gov>

The use of /tmp might be set in the R code of the SwiftR package; will need to check and get back to you.

- Mike

----- Original Message -----
> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> To: swift-user at ci.uchicago.edu
> Sent: Wednesday, June 13, 2012 10:51:32 AM
> Subject: [Swift-user] Question about swiftR temporary and work directories
> We are using swift R. Since we are working on a machine where the
> compute nodes see only the /lustre file system, we need to be able to
> set all the temporary directories there.
> We know how to do that with swift, but swiftR doesn't seem to work
> that way.
> 
> This is the command:
> 
> > library(Swift)
> > swiftInit(server="custom", tc.file="ROC.tc", cf.file="ROC.cf",
> > sites.file="ROCSites.xml")
> Working in /tmp/lpesce/SwiftR/swift.ZKHU
> Running in /tmp/lpesce/SwiftR/swift.ZKHU (linked to
> /tmp/lpesce/SwiftR/swift.custom)
> Started worker manager with pid 10153
> 
> I would like both of those to point to /lustre/beagle/`whoami`/<some
> path>
> where I can decide where to send them or at least that the place is
> under my /lustre folder.
> 
> Any ideas?
> 
> We set these variables before running R:
> module load swift
> export SWIFT_HEAP_MAX=1024M
> export USER_HOME=/lustre/beagle/GCNet
> export SWIFT_HOME=/soft/swift/0.93/
> 
> # set temporary directory references for R, currdir is
> /lustre/beagle/`whoami`
> export TMP=$currdir/tmp
> export TMPDIR=$TMP
> export TEMP=$TMP
> mkdir -p $TMP
> 
> 
> This is the sites file:
> less ROCSites.xml
> <config>
> <pool handle="pbs">
> <execution provider="coaster" jobmanager="local:pbs"/>
> <profile namespace="globus" key="project">CI-CCR000070</profile>
> 
> 
> <profile namespace="globus"
> key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
> 
> <profile namespace="globus" key="jobsPerNode">24</profile>
> <profile namespace="globus" key="maxTime">1800</profile>
> <profile namespace="globus" key="maxwalltime">0:2:00</profile>
> <profile namespace="globus" key="lowOverallocation">100</profile>
> <profile namespace="globus" key="highOverallocation">100</profile>
> 
> <profile namespace="globus" key="slots">1</profile>
> <profile namespace="globus" key="nodeGranularity">1</profile>
> <profile namespace="globus" key="maxNodes">1</profile>
> 
> <profile namespace="karajan" key="jobThrottle">.63</profile>
> <profile namespace="karajan" key="initialScore">10000</profile>
> 
> <filesystem provider="local"/>
> <workdirectory>/lustre/beagle/lpesce/ROCSwift/swift.workdir</workdirectory>
> </pool>
> </config>
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Sun Jun 24 21:53:04 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 24 Jun 2012 21:53:04 -0500 (CDT)
Subject: [Swift-user] Question about swiftR temporary and work
 directories
In-Reply-To: <2006547794.19762.1340592430721.JavaMail.root@zimbra.anl.gov>
Message-ID: <1410224379.19766.1340592784046.JavaMail.root@zimbra.anl.gov>

Lorenzo, I see now that Tim answered this. Is SwiftR now working for you for this problem?

- Mike


----- Original Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "Lorenzo Pesce" <lpesce at uchicago.edu>
> Cc: swift-user at ci.uchicago.edu
> Sent: Sunday, June 24, 2012 9:47:10 PM
> Subject: Re: [Swift-user] Question about swiftR temporary and work directories
> The use of /tmp might be set in the R code of the SwiftR package; will
> need to check and get back to you.
> 
> - Mike
> 
> ----- Original Message -----
> > From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> > To: swift-user at ci.uchicago.edu
> > Sent: Wednesday, June 13, 2012 10:51:32 AM
> > Subject: [Swift-user] Question about swiftR temporary and work
> > directories
> > We are using swift R. Since we are working on a machine where the
> > compute nodes see only the /lustre file system, we need to be able
> > to
> > set all the temporary directories there.
> > We know how to do that with swift, but swiftR doesn't seem to work
> > that way.
> >
> > This is the command:
> >
> > > library(Swift)
> > > swiftInit(server="custom", tc.file="ROC.tc", cf.file="ROC.cf",
> > > sites.file="ROCSites.xml")
> > Working in /tmp/lpesce/SwiftR/swift.ZKHU
> > Running in /tmp/lpesce/SwiftR/swift.ZKHU (linked to
> > /tmp/lpesce/SwiftR/swift.custom)
> > Started worker manager with pid 10153
> >
> > I would like both of those to point to /lustre/beagle/`whoami`/<some
> > path>
> > where I can decide where to send them or at least that the place is
> > under my /lustre folder.
> >
> > Any ideas?
> >
> > We set these variables before running R:
> > module load swift
> > export SWIFT_HEAP_MAX=1024M
> > export USER_HOME=/lustre/beagle/GCNet
> > export SWIFT_HOME=/soft/swift/0.93/
> >
> > # set temporary directory references for R, currdir is
> > /lustre/beagle/`whoami`
> > export TMP=$currdir/tmp
> > export TMPDIR=$TMP
> > export TEMP=$TMP
> > mkdir -p $TMP
> >
> >
> > This is the sites file:
> > less ROCSites.xml
> > <config>
> > <pool handle="pbs">
> > <execution provider="coaster" jobmanager="local:pbs"/>
> > <profile namespace="globus" key="project">CI-CCR000070</profile>
> >
> >
> > <profile namespace="globus"
> > key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
> >
> > <profile namespace="globus" key="jobsPerNode">24</profile>
> > <profile namespace="globus" key="maxTime">1800</profile>
> > <profile namespace="globus" key="maxwalltime">0:2:00</profile>
> > <profile namespace="globus" key="lowOverallocation">100</profile>
> > <profile namespace="globus" key="highOverallocation">100</profile>
> >
> > <profile namespace="globus" key="slots">1</profile>
> > <profile namespace="globus" key="nodeGranularity">1</profile>
> > <profile namespace="globus" key="maxNodes">1</profile>
> >
> > <profile namespace="karajan" key="jobThrottle">.63</profile>
> > <profile namespace="karajan" key="initialScore">10000</profile>
> >
> > <filesystem provider="local"/>
> > <workdirectory>/lustre/beagle/lpesce/ROCSwift/swift.workdir</workdirectory>
> > </pool>
> > </config>
> >
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From lpesce at uchicago.edu  Mon Jun 25 09:58:26 2012
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Mon, 25 Jun 2012 09:58:26 -0500
Subject: [Swift-user] Question about swiftR temporary and work
	directories
In-Reply-To: <1410224379.19766.1340592784046.JavaMail.root@zimbra.anl.gov>
References: <1410224379.19766.1340592784046.JavaMail.root@zimbra.anl.gov>
Message-ID: <3C693343-F33D-4B43-B9AF-57C6B8B3784D@uchicago.edu>

I was on paternity leave and lost a bit track of things ;-)

Let me get back to you soon. The last  decision was that swiftR was not capable of handling this type of problem and that we were most likely going to use Swift or Bash + R.


On Jun 24, 2012, at 9:53 PM, Michael Wilde wrote:

> Lorenzo, I see now that Tim answered this. Is SwiftR now working for you for this problem?
> 
> - Mike
> 
> 
> ----- Original Message -----
>> From: "Michael Wilde" <wilde at mcs.anl.gov>
>> To: "Lorenzo Pesce" <lpesce at uchicago.edu>
>> Cc: swift-user at ci.uchicago.edu
>> Sent: Sunday, June 24, 2012 9:47:10 PM
>> Subject: Re: [Swift-user] Question about swiftR temporary and work directories
>> The use of /tmp might be set in the R code of the SwiftR package; will
>> need to check and get back to you.
>> 
>> - Mike
>> 
>> ----- Original Message -----
>>> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
>>> To: swift-user at ci.uchicago.edu
>>> Sent: Wednesday, June 13, 2012 10:51:32 AM
>>> Subject: [Swift-user] Question about swiftR temporary and work
>>> directories
>>> We are using swift R. Since we are working on a machine where the
>>> compute nodes see only the /lustre file system, we need to be able
>>> to
>>> set all the temporary directories there.
>>> We know how to do that with swift, but swiftR doesn't seem to work
>>> that way.
>>> 
>>> This is the command:
>>> 
>>>> library(Swift)
>>>> swiftInit(server="custom", tc.file="ROC.tc", cf.file="ROC.cf",
>>>> sites.file="ROCSites.xml")
>>> Working in /tmp/lpesce/SwiftR/swift.ZKHU
>>> Running in /tmp/lpesce/SwiftR/swift.ZKHU (linked to
>>> /tmp/lpesce/SwiftR/swift.custom)
>>> Started worker manager with pid 10153
>>> 
>>> I would like both of those to point to /lustre/beagle/`whoami`/<some
>>> path>
>>> where I can decide where to send them or at least that the place is
>>> under my /lustre folder.
>>> 
>>> Any ideas?
>>> 
>>> We set these variables before running R:
>>> module load swift
>>> export SWIFT_HEAP_MAX=1024M
>>> export USER_HOME=/lustre/beagle/GCNet
>>> export SWIFT_HOME=/soft/swift/0.93/
>>> 
>>> # set temporary directory references for R, currdir is
>>> /lustre/beagle/`whoami`
>>> export TMP=$currdir/tmp
>>> export TMPDIR=$TMP
>>> export TEMP=$TMP
>>> mkdir -p $TMP
>>> 
>>> 
>>> This is the sites file:
>>> less ROCSites.xml
>>> <config>
>>> <pool handle="pbs">
>>> <execution provider="coaster" jobmanager="local:pbs"/>
>>> <profile namespace="globus" key="project">CI-CCR000070</profile>
>>> 
>>> 
>>> <profile namespace="globus"
>>> key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
>>> 
>>> <profile namespace="globus" key="jobsPerNode">24</profile>
>>> <profile namespace="globus" key="maxTime">1800</profile>
>>> <profile namespace="globus" key="maxwalltime">0:2:00</profile>
>>> <profile namespace="globus" key="lowOverallocation">100</profile>
>>> <profile namespace="globus" key="highOverallocation">100</profile>
>>> 
>>> <profile namespace="globus" key="slots">1</profile>
>>> <profile namespace="globus" key="nodeGranularity">1</profile>
>>> <profile namespace="globus" key="maxNodes">1</profile>
>>> 
>>> <profile namespace="karajan" key="jobThrottle">.63</profile>
>>> <profile namespace="karajan" key="initialScore">10000</profile>
>>> 
>>> <filesystem provider="local"/>
>>> <workdirectory>/lustre/beagle/lpesce/ROCSwift/swift.workdir</workdirectory>
>>> </pool>
>>> </config>
>>> 
>>> 
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> 
>> --
>> Michael Wilde
>> Computation Institute, University of Chicago
>> Mathematics and Computer Science Division
>> Argonne National Laboratory
>> 
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 



From wilde at mcs.anl.gov  Thu Jun 28 07:12:04 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 28 Jun 2012 07:12:04 -0500 (CDT)
Subject: [Swift-user] Question about swiftR temporary and work
 directories
In-Reply-To: <3C693343-F33D-4B43-B9AF-57C6B8B3784D@uchicago.edu>
Message-ID: <1942023671.25890.1340885524084.JavaMail.root@zimbra.anl.gov>

Hi Lorenzo,

Did you want to meet today? If not, I'll go to Argonne as I have a user there that needs some help. But Im happy to come to campus if I can be of help with your user engagements.

Regards,

- Mike

----- Original Message -----
> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: swift-user at ci.uchicago.edu
> Sent: Monday, June 25, 2012 9:58:26 AM
> Subject: Re: [Swift-user] Question about swiftR temporary and work directories
> I was on paternity leave and lost a bit track of things ;-)
> 
> Let me get back to you soon. The last decision was that swiftR was not
> capable of handling this type of problem and that we were most likely
> going to use Swift or Bash + R.
> 
> 
> On Jun 24, 2012, at 9:53 PM, Michael Wilde wrote:
> 
> > Lorenzo, I see now that Tim answered this. Is SwiftR now working for
> > you for this problem?
> >
> > - Mike
> >
> >
> > ----- Original Message -----
> >> From: "Michael Wilde" <wilde at mcs.anl.gov>
> >> To: "Lorenzo Pesce" <lpesce at uchicago.edu>
> >> Cc: swift-user at ci.uchicago.edu
> >> Sent: Sunday, June 24, 2012 9:47:10 PM
> >> Subject: Re: [Swift-user] Question about swiftR temporary and work
> >> directories
> >> The use of /tmp might be set in the R code of the SwiftR package;
> >> will
> >> need to check and get back to you.
> >>
> >> - Mike
> >>
> >> ----- Original Message -----
> >>> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> >>> To: swift-user at ci.uchicago.edu
> >>> Sent: Wednesday, June 13, 2012 10:51:32 AM
> >>> Subject: [Swift-user] Question about swiftR temporary and work
> >>> directories
> >>> We are using swift R. Since we are working on a machine where the
> >>> compute nodes see only the /lustre file system, we need to be able
> >>> to
> >>> set all the temporary directories there.
> >>> We know how to do that with swift, but swiftR doesn't seem to work
> >>> that way.
> >>>
> >>> This is the command:
> >>>
> >>>> library(Swift)
> >>>> swiftInit(server="custom", tc.file="ROC.tc", cf.file="ROC.cf",
> >>>> sites.file="ROCSites.xml")
> >>> Working in /tmp/lpesce/SwiftR/swift.ZKHU
> >>> Running in /tmp/lpesce/SwiftR/swift.ZKHU (linked to
> >>> /tmp/lpesce/SwiftR/swift.custom)
> >>> Started worker manager with pid 10153
> >>>
> >>> I would like both of those to point to
> >>> /lustre/beagle/`whoami`/<some
> >>> path>
> >>> where I can decide where to send them or at least that the place
> >>> is
> >>> under my /lustre folder.
> >>>
> >>> Any ideas?
> >>>
> >>> We set these variables before running R:
> >>> module load swift
> >>> export SWIFT_HEAP_MAX=1024M
> >>> export USER_HOME=/lustre/beagle/GCNet
> >>> export SWIFT_HOME=/soft/swift/0.93/
> >>>
> >>> # set temporary directory references for R, currdir is
> >>> /lustre/beagle/`whoami`
> >>> export TMP=$currdir/tmp
> >>> export TMPDIR=$TMP
> >>> export TEMP=$TMP
> >>> mkdir -p $TMP
> >>>
> >>>
> >>> This is the sites file:
> >>> less ROCSites.xml
> >>> <config>
> >>> <pool handle="pbs">
> >>> <execution provider="coaster" jobmanager="local:pbs"/>
> >>> <profile namespace="globus" key="project">CI-CCR000070</profile>
> >>>
> >>>
> >>> <profile namespace="globus"
> >>> key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
> >>>
> >>> <profile namespace="globus" key="jobsPerNode">24</profile>
> >>> <profile namespace="globus" key="maxTime">1800</profile>
> >>> <profile namespace="globus" key="maxwalltime">0:2:00</profile>
> >>> <profile namespace="globus" key="lowOverallocation">100</profile>
> >>> <profile namespace="globus" key="highOverallocation">100</profile>
> >>>
> >>> <profile namespace="globus" key="slots">1</profile>
> >>> <profile namespace="globus" key="nodeGranularity">1</profile>
> >>> <profile namespace="globus" key="maxNodes">1</profile>
> >>>
> >>> <profile namespace="karajan" key="jobThrottle">.63</profile>
> >>> <profile namespace="karajan" key="initialScore">10000</profile>
> >>>
> >>> <filesystem provider="local"/>
> >>> <workdirectory>/lustre/beagle/lpesce/ROCSwift/swift.workdir</workdirectory>
> >>> </pool>
> >>> </config>
> >>>
> >>>
> >>> _______________________________________________
> >>> Swift-user mailing list
> >>> Swift-user at ci.uchicago.edu
> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >>
> >> --
> >> Michael Wilde
> >> Computation Institute, University of Chicago
> >> Mathematics and Computer Science Division
> >> Argonne National Laboratory
> >>
> >> _______________________________________________
> >> Swift-user mailing list
> >> Swift-user at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From iraicu at cs.iit.edu  Thu Jun 28 13:52:23 2012
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Thu, 28 Jun 2012 13:52:23 -0500
Subject: [Swift-user] eScience Conference submission extension - 18 July
Message-ID: <4FECA7E7.9050805@cs.iit.edu>

Due to numerous requests, we are extended the paper deadline, and 
dropping the abstract deadline completely:


CALL FOR PAPERS

8th IEEE International Conference on eScience
http://www.ci.uchicago.edu/escience2012/
October 8-12, 2012
Chicago, IL, USA

Researchers in all disciplines are increasingly adopting digital tools, 
techniques and practices, often in communities and projects that span 
disciplines, laboratories, organizations, and national boundaries. The 
eScience 2012 conference is designed to bring together leading 
international and interdisciplinary research communities, developers, 
and users of eScience applications and enabling IT technologies. The 
conference serves as a forum to present the results of the latest 
applications research and product/tool developments and to highlight 
related activities from around the world. Also, we are now entering the 
second decade of eScience and the 2012 conference gives an opportunity 
to take stock of what has been achieved so far and look forward to the 
challenges and opportunities the next decade will bring.

A special emphasis of the 2012 conference is on advances in the 
application of technology in a particular discipline. Accordingly, 
significant advances in applications science and technology will be 
considered as important as the development of new technologies 
themselves. Further, we welcome contributions in educational activities 
under any of these disciplines.

As a result, the conference will be structured around two e-Science tracks:

. eScience Algorithms and Applications
. eScience application areas, including:
. Physical sciences
. Biomedical sciences
. Social sciences and humanities
. Data-oriented approaches and applications
. Compute-oriented approaches and applications
. Extreme scale approaches and applications
. Cyberinfrastructure to support eScience
. Novel hardware
. Novel uses of production infrastructure
. Software and services
. Tools

The conference proceedings will be published by the IEEE Computer 
Society Press, USA and will be made available online through the IEEE 
Digital Library. Selected papers will be invited to submit extended 
versions to a special issue of the Future Generation Computer Systems 
(FGCS)journal.

SUBMISSION PROCESS
Authors are invited to submit papers with unpublished, original work of 
not more than 8 pages of double column text using single spaced 10 point 
size on 8.5 x 11 inch pages, as per IEEE 8.5 x 11 manuscript guidelines. 
(Up to 2 additional pages may be purchased for US$150/page)

Templates are available from 
http://www.ieee.org/conferences_events/conferences/publishing/templates.html.

Authors should submit a PDF file that will print on a PostScript printer 
tohttps://www.easychair.org/conferences/?conf=escience2012

(Note that paper submitters also must submit an abstract in advance of 
the paper deadline. This should be done through the same site where 
papers are submitted.)

It is a requirement that at least one author of each accepted paper 
attend the conference.

IMPORTANT DATES

Abstract submission no longer required
Paper submission: extended to 18 July 2012 (firm)
Paper author notification: 22 August 2012
Camera-ready papers due: 10 September 2012
Conference: 8-12 October 2012




In addition to the eScience conference itself, there are six associated 
workshops and one tutorial 
(http://www.ci.uchicago.edu/escience2012/workshops.php)

  * Extending High-Performance Computing Beyond its Traditional User
    Communities, http://www.psc.edu/events/escience-2012-workshop/
  * 2nd International Workshop on Analyzing and Improving Collaborative
    eScience with Social Networks (eSoN 12),
    http://www.ci.uchicago.edu/eson2012/
  * Advances in eHealth,
    http://www.scalalife.eu/content/advances-ehealth-2012-workshop
  * Maintainable Software Practices in e-Science,
    http://software.ac.uk/maintainable-software-practice-workshop
  * eScience Meets the Instrument,
    https://confluence-vre.its.monash.edu.au/display/escience2012/eScience+Meets+the+Instrument
  * Collaborative research using eScience infrastructure and high speed
    networks,
    http://www.surfnet.nl/en/Hybride_netwerk/SURFlichtpaden/Pages/CollaborativeresearchusingeScienceinfrastructureandhighspeednetworks.aspx


  * Tutorial: Big Data Processing: Lessons from Industry and
    Applications in Science,
    http://www.ci.uchicago.edu/escience2012/tutorial.php






CONFERENCE ORGANIZATION

General Chair
. Ian Foster, University of Chicago & Argonne National Laboratory, USA
Program Co-Chairs
. Daniel S. Katz, University of Chicago & Argonne National Laboratory, USA
. Heinz Stockinger, SIB Swiss Institute of Bioinformatics, Switzerland
Program Vice Co-Chairs
. eScience Algorithms and Applications Track
. David Abramson, Monash University, Australia
. Gabrielle Allen, Louisiana State University, USA
. Cyberinfrastructure to support eScience Track
. Rosa M. Badia, Barcelona Supercomputing Center / CSIC, Spain
. Geoffrey Fox, Indiana University, USA
Early Results and Works-in-Progress Posters Chair
. Roger Barga, Microsoft, USA
Workshops Chair
. Ruth Pordes, FNAL, USA
Sponsorship Chair
. Charlie Catlett, Argonne National Laboratory, USA
Conference Manager and Finance Chair
. Julie Wulf-Knoerzer, University of Chicago & Argonne National 
Laboratory, USA
Publicity Chairs
. Kento Aida, National Institute of Informatics, Japan
. Ioan Raicu, Illinois Institute of Technology, USA
. David Wallom, Oxford e-Research Centre, UK
Local Organizing Committee
. Ninfa Mayorga, University of Chicago, USA
. Evelyn Rayburn, University of Chicago, USA
. Lynn Valentini, Argonne National Laboratory, USA
Program Committee
. eScience Algorithms and Applications Track
. Srinivas Aluru, Iowa State University, USA
. Ashiq Anjum, University of Derby, UK
. David A. Bader, Georgia Institute of Technology, USA
. Jon Blower, University of Reading, UK
. Paul Bonnington, Monash University, Australia
. Simon Cox, University of Southampton, UK
. David De Roure, Oxford e-Research Centre, UK
. George Djorgovski, California Institute of Technology, USA
. Anshu Dubey, University of Chicago & Argonne National Laboratory, USA
. Yuri Estrin, Monash University, Australia
. Dan Fay, Microsoft, USA
. Jeremy Frey, University of Southampton, UK
. Wolfgang Gentzsch, HPC Consultant, Germany
. Lutz Gross, The University of Queensland, Austrialia
. Sverker Holmgren, Uppsala University, Sweden
. Bill Howe, University of Washington, USA
. Marina Jirotka, University of Oxford, UK
. Timoleon Kipouros, University of Cambridge, UK
. Kerstin Kleese van Dam, Pacific Northwest National Laboratory, USA
. Arun S. Konagurthu, Monash University, Australia
. Peter Kunszt, SystemsX.ch <http://SystemsX.ch/>, Switzerland
. Alexey Lastovetsky, University College Dublin, Ireland
. Andrew Lewis, Griffith University, Australia
. Sergio Maffioletti, University of Zurich, Switzerland
. Amitava Majumdar, San Diego Supercomputer Center, University of 
California at San Diego, USA
. Rui Mao, Shenzhen University, China
. Madhav V. Marathe, Virginia Tech, USA
. Maryann Martone, University of California at San Diego, USA
. Louis Moresi, Monash University, Australia
. Riccardo Murri, University of Zurich, Switzerland
. Silvia D. Olabarriaga, Academic Medical Center of the University of 
Amsterdam, Netherlands
. Enrique S. Quintana-Ort?, Universidad Jaume I, Spain
. Abani Patra, University at Buffalo, USA
. Rob Pennington, NSF, USA
. Andrew Perry, Monash University, Australia
. Beth Plale, Indiana University, USA
. Michael Resch, University of Stuttgart, Germany
. Adrian Sandu, Virginia Tech, USA
. Mark Savill, Cranfield University, UK
. Erik Schnetter, Perimeter Institute for Theoretical Physics, Canada
. Edward Seidel, Louisiana State University, USA
. Suzanne M. Shontz, The Pennsylvania State University, USA
. David Skinner, Lawrence Berkeley National Laboratory, USA
. Alan Sussman, University of Maryland, USA
. Alex Szalay, Johns Hopkins University, USA
. Domenico Talia, ICAR-CNR & University of Calabria, Italy
. Jian Tao, Louisiana State University, USA
. David Wallom, Oxford e-Research Centre, UK
. Shaowen Wang, University of Illinois at Urbana-Champaign, USA
. Michael Wilde, Argonne National Laboratory & University of Chicago, USA
. Nancy Wilkins-Diehr, San Diego Supercomputer Center, University of 
California at San Diego, USA
. Wu Zhang, Shanghai University, China
. Yunquan Zhang, Chinese Academy of Sciences, China
. Cyberinfrastructure to support eScience Track
. Deb Agarwal, Lawrence Berkeley National Laboratory, USA
. Ilkay Altintas, San Diego Supercomputer Center, University of 
California at San Diego, USA
. Henri Bal, Vrije Universiteit, Netherlands
. Roger Barga, Microsoft, USA
. Martin Berzins, University of Utah, USA
. John Brooke, University of Manchester, UK
. Thomas Fahringer, University of Innsbruck, Austria
. Gilles Fedak, INRIA, France
. Jos? A. B. Fortes, University of Florida, USA
. Yolanda Gil, ISI/USC, USA
. Madhusudhan Govindaraju, SUNY Binghamton, USA
. Thomas Hacker, Purdue University, USA
. Ken Hawick, Massey University, New Zealand
. Marty Humphrey, University of Virginia, USA
. Hai Jin, Huazhong University of Science and Technology, China
. Thilo Kielmann, Vrije Universiteit, Netherlands
. Scott Klasky, Oak Ridge National Laboratory, USA
. Isao Kojima, AIST, Japan
. Tevfik Kosar, University at Buffalo, USA
. Dieter Kranzlmueller, LMU & LRZ Munich, Germany
. Erwin Laure, KTH, Sweden
. Jysoo Lee, KISTI, Korea
. Li Xiaoming, Peking University, China
. Bertram Lud?scher, University of California, Davis, USA
. Andrew Lumsdaine, Indiana University, USA
. Tanu Malik, University of Chicago, USA
. Satoshi Matsuoka, Tokyo Institute of Technology, Japan
. Reagan Moore, University of North Carolina at Chapel Hill, USA
. Shirley Moore, University of Kentucky, USA
. Steven Newhouse, EGI, Netherlands
. Dhabaleswar K. (DK) Panda, The Ohio State University, USA
. Manish Parashar, Rutgers University, USA
. Ron Perrott, University of Oxford, UK
. Depei Qian, Beihang University, China
. Judy Qui, Indiana University, USA
. Ioan Raicu, Illinois Institute of Technology, USA
. Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA
. Omer Rana, Cardiff University, UK
. Paul Roe, Queensland University of Technology, Australia
. Bruno Schulze, LNCC, Brazil
. Marc Snir, Argonne National Laboratory & University of Illinois at 
Urbana-Champaign, USA
. Xian-He Sun, Illinois Institute of Technology, USA
. Yoshio Tanaka, AIST, Japan
. Michela Taufer, University of Delaware, USA
. Kerry Taylor, CSIRO, Australia
. Douglas Thain, University of Notre Dame, USA
. Paul Watson, Newcastle University, UK
. Jian Zhang, Northern Illinois University, USA
. Jun Zhao, University of Oxford, UK

Sponsors:
. University of Chicago
. Argonne National Laboratory
. IEEE
. CSIRO
. Indiana University
. additional sponsorship opportunities are available

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120628/fd6a47c9/attachment.html>

From kazutaka at gmail.com  Sat Jun  9 23:53:28 2012
From: kazutaka at gmail.com (Kazutaka Takahashi)
Date: Sun, 10 Jun 2012 04:53:28 -0000
Subject: [Swift-user] [Swift-devel] ram disk with swift
In-Reply-To: <1272810551.111958.1339191283314.JavaMail.root@zimbra-mb2.anl.gov>
References: <B976B06A-D7C4-4017-9CD4-B3C3F2493739@uchicago.edu>
	<1272810551.111958.1339191283314.JavaMail.root@zimbra-mb2.anl.gov>
Message-ID: <CAK8C4JW4Hm7seCz-4jU6Zy=q2hp8DwDvZh23ovpMM49B1V=eww@mail.gmail.com>

Sorry for a late reply and thank you for all!
Lorenzo and I will go over modifications on monday and will test out those
codes.
Thanks and enjoy the rest of the weekend!
Taka


On Fri, Jun 8, 2012 at 4:34 PM, David Kelly <davidk at ci.uchicago.edu> wrote:

> Here is a portion of a script that that copies some frequently used data
> to a node's shared memory. I think it is pretty similar to the script that
> Glen sent earlier.
>
> dssatdir=/dev/shm/dssat
> dssattmp=/dev/shm/dssat.tmp
>
> if [ ! -d $dssatdir ]; then
>  if mkdir $dssattmp 2>/dev/null; then
>    echo $$ copying
>    savewd=$PWD
>    cd $dssattmp
>
>    mkdir -p refdata
>    cp $refdata/* refdata/
>
>    mkdir -p campaign
>    cp $campaign/*.MZX campaign/
>
>    mkdir -p binpath
>    cp $binpath/*.EXE binpath/
>
>    cd $savewd
>    mv $dssattmp $dssatdir
>  else
>    while [ ! -d $dssatdir ]; do
>     # echo $$ sleeping
>      sleep 1;
>    done
>  fi
> fi
>
>
>
> ----- Original Message -----
> > From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> > To: "Michael Wilde" <wilde at mcs.anl.gov>, "Swift Devel" <
> swift-devel at ci.uchicago.edu>, swift-user at ci.uchicago.edu
> > Cc: "Kazutaka Takahashi" <kazutaka at gmail.com>
> > Sent: Friday, June 8, 2012 2:39:19 PM
> > Subject: [Swift-devel] ram disk with swift
> > Mike,
> > We were talking about putting the input files for Taka's calculations
> > (the X.mat files), which are rather small ~ 60Kb or so, in ram disk
> > and try to use them for all the apps (that use the same file). giving
> > the size, we could even load up all of them for each node
> > (7*3*60=1.2MB, that is zero) to reduce traffic towards /lustre and
> > thus avoid the meta-data slow downs that Beagle seems to have.
> >
> > If we copy files in /tmp (as far as I understand it would be ramdisk)
> > on the each node through swift, would it be there for all the apps?
> >
> > Thanks,
> >
> > Lorenzo
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
What is essential is invisible to the eye
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120610/1701d93c/attachment.html>