[Swift-commit] r7167 - SwiftApps/Swift-MapRed/mapred_combiner_V4
yadunandb at ci.uchicago.edu
yadunandb at ci.uchicago.edu
Wed Oct 16 11:14:02 CDT 2013
Author: yadunandb
Date: 2013-10-16 11:14:02 -0500 (Wed, 16 Oct 2013)
New Revision: 7167
Removed:
SwiftApps/Swift-MapRed/mapred_combiner_V4/combiner.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/combiner_template.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/compile
SwiftApps/Swift-MapRed/mapred_combiner_V4/mapper.sh
Modified:
SwiftApps/Swift-MapRed/mapred_combiner_V4/LAB_RECORDS
SwiftApps/Swift-MapRed/mapred_combiner_V4/beagle.xml
SwiftApps/Swift-MapRed/mapred_combiner_V4/mapper_template.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/multiple.xml
SwiftApps/Swift-MapRed/mapred_combiner_V4/reducer_template.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/run.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/setup.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/swiftreduce.swift
SwiftApps/Swift-MapRed/mapred_combiner_V4/tc.data
Log:
minor updates
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/LAB_RECORDS
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/LAB_RECORDS 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/LAB_RECORDS 2013-10-16 16:14:02 UTC (rev 7167)
@@ -4426,3 +4426,58 @@
ACTUAL COUNT :
EXPECTED COUNT: 100000
---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:20 X FILESIZE:10
+Warning: Parameter reducer, on line 47, shadows variable of same name on line 30
+Warning: Parameter all_map, on line 47, shadows variable of same name on line 43
+Warning: Parameter reducer, on line 51, shadows variable of same name on line 30
+Warning: Parameter all_map, on line 51, shadows variable of same name on line 43
+Warning: Parameter reducer, on line 55, shadows variable of same name on line 30
+chunks 20 filesize 10 site multiple type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 2000000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:10 X FILESIZE:1
+Warning: Parameter reducer, on line 47, shadows variable of same name on line 30
+Warning: Parameter all_map, on line 47, shadows variable of same name on line 43
+Warning: Parameter reducer, on line 51, shadows variable of same name on line 30
+Warning: Parameter all_map, on line 51, shadows variable of same name on line 43
+Warning: Parameter reducer, on line 55, shadows variable of same name on line 30
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1515-psg21p04
+Progress: time: Wed, 16 Oct 2013 15:15:06 +0000
+Progress: time: Wed, 16 Oct 2013 15:15:23 +0000 Submitting:4 Submitted:1 Finished successfully:5
+Progress: time: Wed, 16 Oct 2013 15:15:36 +0000 Submitted:5 Finished successfully:5
+Progress: time: Wed, 16 Oct 2013 15:16:06 +0000 Submitted:5 Finished successfully:5
+chunks 10 filesize 1 site multiple type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 100000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:10 X FILESIZE:1
+Warning: Parameter reducer, on line 47, shadows variable of same name on line 30
+Warning: Parameter all_map, on line 47, shadows variable of same name on line 43
+Warning: Parameter reducer, on line 51, shadows variable of same name on line 30
+Warning: Parameter all_map, on line 51, shadows variable of same name on line 43
+Warning: Parameter reducer, on line 55, shadows variable of same name on line 30
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1516-3xtve80d
+Progress: time: Wed, 16 Oct 2013 15:16:44 +0000
+Progress: time: Wed, 16 Oct 2013 15:16:50 +0000 Stage out:1 Finished successfully:11
+Final status: Wed, 16 Oct 2013 15:16:50 +0000 Finished successfully:13
+real 9.12
+user 16.47
+sys 2.72
+chunks 10 filesize 1 site multiple type Swift+Chirp
+---------------------------------------------
+COUNT matches expected records
+ACTUAL COUNT : 100000
+EXPECTED COUNT: 100000
+---------------------------------------------
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/beagle.xml
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/beagle.xml 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/beagle.xml 2013-10-16 16:14:02 UTC (rev 7167)
@@ -11,7 +11,7 @@
<profile namespace="globus" key="maxtime">2400</profile>
<profile namespace="globus" key="maxWalltime">00:20:00</profile>
<profile namespace="globus" key="queue">batch</profile>
- <profile namespace="globus" key="userHomeOverride">/lustre/beagle/yadunandb/swiftwork</profile>
+ <profile namespace="globus" key="userHomeOverride">/lustre/beagle/BEAGLE_USERNAME/swiftwork</profile>
<profile namespace="globus" key="slots">10</profile>
<profile namespace="globus" key="maxnodes">1</profile>
<profile namespace="globus" key="nodeGranularity">1</profile>
@@ -19,6 +19,6 @@
<profile namespace="karajan" key="initialScore">10000</profile>
<!-- <filesystem provider="local"/> -->
<!-- <workdirectory>/lustre/beagle/yadunandb/swiftwork</workdirectory> -->
- <workdirectory>/tmp/yadunandb/swiftwork</workdirectory>
+ <workdirectory>/tmp/BEAGLE_USERNAME/swiftwork</workdirectory>
</pool>
</config>
Deleted: SwiftApps/Swift-MapRed/mapred_combiner_V4/combiner.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/combiner.sh 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/combiner.sh 2013-10-16 16:14:02 UTC (rev 7167)
@@ -1,40 +0,0 @@
-#!/bin/bash
-
-ARGS=($*)
-SUM=0
-COUNT=0
-
-# When strings in format node files... are passed to combiner
-# 1. Run a local_combiner on the remote node
-# 2. Return a file pointer
-NODE=${ARGS[0]}
-FILES=(${ARGS[*]:1})
-REMOTE_PORT=29900
-HOSTNAME=$(hostname -f)
-
-############## LOCAL COMBINER CODE ###################
-cat << EOF > $NODE.sh
-#!/bin/bash
-ID=$RANDOM
-FILES=(${FILES[*]})
-EOF
-cat <<'EOF' >> $NODE.sh
-#USER_REDUCE
-reduce(){
- FILES=($*)
- DIRNAME=$(dirname ${FILES[0]})
- RESULT=$DIRNAME/$(hostname -f).$RANDOM.imd
- awk '{ sum += $1; count += $2 } END { print sum,count }' ${FILES[*]} > $RESULT
- echo $RESULT
- rm -rf ${FILES[*]} &> /dev/null
-}
-echo -e "$(hostname -f) $(reduce ${FILES[*]})"
-EOF
-chmod a+x $NODE.sh
-############# END LOCAL COMBINER #####################
-cp /lustre/beagle/yadunandb/bin/exec_client .
-echo "Combiner ${ARGS[*]}" 1>&2
-echo "Combiner [./exec_client $NODE $NODE.sh]" 1>&2
-cat $NODE.sh 1>&2
-./exec_client $NODE $NODE.sh
-exit 0
Deleted: SwiftApps/Swift-MapRed/mapred_combiner_V4/combiner_template.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/combiner_template.sh 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/combiner_template.sh 2013-10-16 16:14:02 UTC (rev 7167)
@@ -1,32 +0,0 @@
-#!/bin/bash
-
-ARGS=($*)
-SUM=0
-COUNT=0
-
-# When strings in format node files... are passed to combiner
-# 1. Run a local_combiner on the remote node
-# 2. Return a file pointer
-NODE=${ARGS[0]}
-FILES=(${ARGS[*]:1})
-REMOTE_PORT=29900
-HOSTNAME=$(hostname -f)
-
-############## LOCAL COMBINER CODE ###################
-cat << EOF > $NODE.sh
-#!/bin/bash
-ID=$RANDOM
-FILES=(${FILES[*]})
-EOF
-cat <<'EOF' >> $NODE.sh
-#USER_REDUCE
-echo -e "$(hostname -f) $(reduce ${FILES[*]})"
-EOF
-chmod a+x $NODE.sh
-############# END LOCAL COMBINER #####################
-cp /lustre/beagle/yadunandb/bin/exec_client .
-echo "Combiner ${ARGS[*]}" 1>&2
-echo "Combiner [./exec_client $NODE $NODE.sh]" 1>&2
-cat $NODE.sh 1>&2
-./exec_client $NODE $NODE.sh
-exit 0
Deleted: SwiftApps/Swift-MapRed/mapred_combiner_V4/compile
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/compile 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/compile 2013-10-16 16:14:02 UTC (rev 7167)
@@ -1,13 +0,0 @@
-#!/bin/bash
-
-gcc server.c -o exec_server
-gcc client.c -o exec_client
-
-chmod a+x exec_server
-chmod a+x exec_client
-
-scp exec_server swift.rcc.uchicago.edu:/scratch/midway/yadunand/bin/
-scp exec_client swift.rcc.uchicago.edu:/scratch/midway/yadunand/bin/
-scp exec_server login4.beagle.ci.uchicago.edu:/lustre/beagle/yadunandb/bin/
-scp exec_client login4.beagle.ci.uchicago.edu:/lustre/beagle/yadunandb/bin/
-
Deleted: SwiftApps/Swift-MapRed/mapred_combiner_V4/mapper.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/mapper.sh 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/mapper.sh 2013-10-16 16:14:02 UTC (rev 7167)
@@ -1,63 +0,0 @@
-#!/bin/bash
-
-# By default with ARG1:100 and SLICESIZE=10000, this script will generate
-# 10^6 records.
-ARG1=1
-[ ! -z $1 ] && ARG1=$1
-
-FILE="input_$RANDOM.txt"
-HOSTNAME=$(hostname -f)
-if [[ "$HOSTNAME" == *midway* ]]; then # On midway node
- EXECSERVER=/scratch/midway/yadunand/bin/exec_server
- LOGFOLDER=/scratch/midway/yadunand
- FOLDER="/dev/shm"
- HOSTNAME="midway_$HOSTNAME"
-
-elif [[ "$HOSTNAME" == *nid* ]]; then # On beagle node
- EXECSERVER=/lustre/beagle/yadunandb/bin/exec_server
- LOGFOLDER=/lustre/beagle/yadunandb/Swift-MapRed/mapred_combiner_multisite
- FOLDER="/dev/shm"
- HOSTNAME="beagle_$HOSTNAME"
-fi
-
-TIMEOUT=1200
-NC_LISTEN_PORT=29900
-
-# Start services on Node
-JOBS_LEVEL=`echo $PWD | grep -o ".*jobs"`
-if [ "$?" == "0" ]
-then
- mkdir $JOBS_LEVEL/CHIRPING
- if [ "$?" == "0" ]
- then
- killall -u $USER chirp_server;
- echo "unix:$USER rwlds" > $FOLDER/acl.conf
- echo "hostname:* rwl" >> $FOLDER/acl.conf
- which chirp_server 1>&2
- timeout $TIMEOUT chirp_server -A $FOLDER/acl.conf -r $FOLDER &
- timeout $TIMEOUT $EXECSERVER &> $LOGFOLDER/exec_server_log-$HOSTNAME.log &
- fi
- FILE=$FOLDER/$FILE;
-else
- FILE=$PWD/$FILE
-fi
-
-#USER_MAP
-mapper (){
- ARG1=$1;
- LOWERLIMIT=0
- UPPERLIMIT=1000000
- SLICESIZE=10000 # 10^4 records padded to 100B would result in 1MB file
- #Generate $SLICESIZE number of 100B records of format: RANDOM_NUMBER 1
- shuf -i $LOWERLIMIT-$UPPERLIMIT -n $SLICESIZE | awk '{printf "%-97s %s\n", $0, 1}' > tmp_shuf
- for i in $(seq 1 1 $ARG1)
- do
- cat tmp_shuf
- done
- rm tmp_shuf
-}
-
-mapper $ARG1 > $FILE
-echo "$HOSTNAME $FILE"
-ps -u $USER 1>&2
-exit 0;
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/mapper_template.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/mapper_template.sh 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/mapper_template.sh 2013-10-16 16:14:02 UTC (rev 7167)
@@ -8,14 +8,14 @@
FILE="input_$RANDOM.txt"
HOSTNAME=$(hostname -f)
if [[ "$HOSTNAME" == *midway* ]]; then # On midway node
- EXECSERVER=/scratch/midway/yadunand/bin/exec_server
- LOGFOLDER=/scratch/midway/yadunand
+ EXECSERVER=/scratch/midway/MIDWAY_USERNAME/bin/exec_server
+ LOGFOLDER=/scratch/midway/MIDWAY_USERNAME
FOLDER="/dev/shm"
HOSTNAME="midway_$HOSTNAME"
elif [[ "$HOSTNAME" == *nid* ]]; then # On beagle node
- EXECSERVER=/lustre/beagle/yadunandb/bin/exec_server
- LOGFOLDER=/lustre/beagle/yadunandb/Swift-MapRed/mapred_combiner_multisite
+ EXECSERVER=/lustre/beagle/BEAGLE_USERNAME/bin/exec_server
+ LOGFOLDER=/lustre/beagle/BEAGLE_USERNAME/Swift-MapRed/mapred_combiner_multisite
FOLDER="/dev/shm"
HOSTNAME="beagle_$HOSTNAME"
fi
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/multiple.xml
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/multiple.xml 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/multiple.xml 2013-10-16 16:14:02 UTC (rev 7167)
@@ -12,7 +12,7 @@
<profile namespace="karajan" key="jobThrottle">0.03</profile>
<profile namespace="karajan" key="initialScore">10000</profile>
<filesystem provider="local"/>
- <workdirectory>/tmp/yadunand</workdirectory>
+ <workdirectory>/tmp/{env.USER}</workdirectory>
</pool>
<pool handle="beagle">
@@ -24,7 +24,7 @@
<profile namespace="globus" key="maxtime">1200</profile>
<profile namespace="globus" key="maxWalltime">00:10:00</profile>
<profile namespace="globus" key="queue">development</profile>
- <profile namespace="globus" key="userHomeOverride">/lustre/beagle/yadunandb/swiftwork</profile>
+ <profile namespace="globus" key="userHomeOverride">/lustre/beagle/BEAGLE_USERNAME/swiftwork</profile>
<profile namespace="globus" key="slots">1</profile>
<profile namespace="globus" key="maxnodes">4</profile>
<profile namespace="globus" key="nodeGranularity">4</profile>
@@ -32,52 +32,10 @@
<profile namespace="karajan" key="initialScore">10000</profile>
<!-- <profile namespace="karajan" key="workerLoggingLevel">trace</profile> -->
<profile namespace="globus" key="workerLoggingLevel">DEBUG</profile>
- <!-- <workdirectory>/lustre/beagle/yadunandb/swiftwork</workdirectory> -->
- <workdirectory>/tmp/yadunandb/swiftwork</workdirectory>
+ <!-- <workdirectory>/lustre/beagle/BEAGLE_USERNAME/swiftwork</workdirectory> -->
+ <workdirectory>/tmp/BEAGLE_USERNAME/swiftwork</workdirectory>
</pool>
-
- <pool handle="beagle_local">
- <execution provider="coaster" jobmanager="local:pbs"/>
- <profile namespace="globus" key="jobsPerNode">1</profile>
- <profile namespace="globus" key="lowOverAllocation">100</profile>
- <profile namespace="globus" key="highOverAllocation">100</profile>
- <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
- <!-- to use a beage reservation, eg:
- <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24;pbs.resource_list=advres=wilde.1768</profile>
- -->
- <profile namespace="globus" key="maxtime">2400</profile>
- <profile namespace="globus" key="maxWalltime">00:20:00</profile>
- <profile namespace="globus" key="queue">batch</profile>
- <profile namespace="globus" key="userHomeOverride">/lustre/beagle/yadunandb/swiftwork</profile>
- <profile namespace="globus" key="slots">1</profile>
- <profile namespace="globus" key="maxnodes">4</profile>
- <profile namespace="globus" key="nodeGranularity">1</profile>
- <profile namespace="karajan" key="jobThrottle">1.00</profile>
- <profile namespace="karajan" key="initialScore">10000</profile>
- <!-- <filesystem provider="local"/> -->
- <!-- <workdirectory>/lustre/beagle/yadunandb/swiftwork</workdirectory> -->
- <workdirectory>/tmp/yadunandb/swiftwork</workdirectory>
- </pool>
-
- <pool handle="beagle_remote">
- <execution provider="coaster" jobmanager="ssh-cl:pbs" url="login4.beagle.ci.uchicago.edu"/>
- <profile namespace="globus" key="jobsPerNode">4</profile>
- <profile namespace="globus" key="lowOverAllocation">100</profile>
- <profile namespace="globus" key="highOverAllocation">100</profile>
- <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24;pbs.resource_list=advres=wilde.1768</profile>
- <profile namespace="globus" key="maxtime">1200</profile>
- <profile namespace="globus" key="maxWalltime">00:10:00</profile>
- <profile namespace="globus" key="userHomeOverride">/lustre/beagle/yadunandb/swiftwork</profile>
- <profile namespace="globus" key="slots">1</profile>
- <profile namespace="globus" key="maxnodes">1</profile>
- <profile namespace="globus" key="nodeGranularity">1</profile>
- <profile namespace="karajan" key="jobThrottle">1.00</profile>
- <profile namespace="karajan" key="initialScore">10000</profile>
- <!-- <workdirectory>/tmp/yadunandb/swiftwork</workdirectory> -->
- <workdirectory>/lustre/beagle/yadunandb/swiftwork</workdirectory>
- </pool>
-
<pool handle="midway">
<execution provider="coaster" jobmanager="local:slurm"/>
<profile namespace="globus" key="queue">westmere</profile>
@@ -93,7 +51,7 @@
<profile namespace="globus" key="nodeGranularity">4</profile>
<profile namespace="karajan" key="jobThrottle">.64</profile>
<profile namespace="karajan" key="initialScore">10000</profile>
- <workdirectory>/tmp/yadunand</workdirectory>
+ <workdirectory>/tmp/MIDWAY_USERNAME</workdirectory>
</pool>
</config>
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/reducer_template.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/reducer_template.sh 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/reducer_template.sh 2013-10-16 16:14:02 UTC (rev 7167)
@@ -11,17 +11,17 @@
ID=$RANDOM
HOSTNAME=$(hostname -f)
if [[ "$HOSTNAME" == *midway* ]]; then # On midway node
- EXECSERVER=/scratch/midway/yadunand/bin/exec_server
- EXECCLIENT=/scratch/midway/yadunand/bin/exec_client
- LOGFOLDER=/scratch/midway/yadunand
+ EXECSERVER=/scratch/midway/MIDWAY_USERNAME/bin/exec_server
+ EXECCLIENT=/scratch/midway/MIDWAY_USERNAME/bin/exec_client
+ LOGFOLDER=/scratch/midway/MIDWAY_USERNAME
FOLDER="/dev/shm"
HOSTID="midway_$HOSTNAME"
SITE="midway"
elif [[ "$HOSTNAME" == *nid* ]]; then # On beagle node
- EXECSERVER=/lustre/beagle/yadunandb/bin/exec_server
- EXECCLIENT=/lustre/beagle/yadunandb/bin/exec_client
- LOGFOLDER=/lustre/beagle/yadunandb/Swift-MapRed/
+ EXECSERVER=/lustre/beagle/BEAGLE_USERNAME/bin/exec_server
+ EXECCLIENT=/lustre/beagle/BEAGLE_USERNAME/bin/exec_client
+ LOGFOLDER=/lustre/beagle/BEAGLE_USERNAME/Swift-MapRed/
FOLDER="/dev/shm"
HOSTID="beagle_$HOSTNAME"
SITE="beagle"
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/run.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/run.sh 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/run.sh 2013-10-16 16:14:02 UTC (rev 7167)
@@ -15,15 +15,7 @@
./clean.sh
-if [ "$SITE" != "beagle" ]
-then
- #./compile
-# cp exec_server /lustre/beagle/yadunandb/bin/
-# cp exec_client /lustre/beagle/yadunandb/bin/
- #cp exec_client /scratch/midway/yadunand/bin/
- #cp exec_server /scratch/midway/yadunand/bin/
- echo "Skipping compile and copy"
-fi;
+
#Call as check_error $? <ERR_CODE> <Error message>
check_error()
{
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/setup.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/setup.sh 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/setup.sh 2013-10-16 16:14:02 UTC (rev 7167)
@@ -1,5 +1,10 @@
#!/bin/bash
+[ ! -z $BEAGLE_USERNAME ] || BEAGLE_USERNAME="yadunandb"
+[ ! -z $MIDWAY_USERNAME ] || MIDWAY_USERNAME="yadunand"
+BEAGLE_SHARE_FOLDER=/lustre/beagle/$BEAGLE_USERNAME/bin/
+MIDWAY_SHARE_FOLDER=/scratch/midway/$MIDWAY_USERNAME/bin/
+
if [ ! -f "map_core.sh" ]
then
echo "map_core.sh missing"
@@ -7,17 +12,17 @@
fi
if [ ! -f "mapper_template.sh" ];
-then
+then
echo "mapper_template missing"
exit 0;
else
sed -e "/\#USER_MAP/r map_core.sh" mapper_template.sh \
> mapper.sh
+ sed -i "s/BEAGLE_USERNAME/$BEAGLE_USERNAME/g" mapper.sh
+ sed -i "s/MIDWAY_USERNAME/$MIDWAY_USERNAME/g" mapper.sh
chmod 777 mapper.sh
fi;
-
-
if [ ! -f "reduce_core.sh" ]
then
echo "reduce_core.sh missing"
@@ -25,27 +30,31 @@
fi;
if [ ! -f "reducer_template.sh" ];
-then
+then
echo "reducer_template missing"
exit 0;
else
sed -e "/\#USER_REDUCE/r reduce_core.sh" reducer_template.sh \
> reducer.sh
+ sed -i "s/BEAGLE_USERNAME/$BEAGLE_USERNAME/g" reducer.sh
+ sed -i "s/MIDWAY_USERNAME/$MIDWAY_USERNAME/g" reducer.sh
chmod 777 reducer.sh
fi;
-if [ ! -f "combiner_template.sh" ];
-then
- echo "combiner_template missing"
- exit 0
-else
- sed -e "/\#USER_REDUCE/r reduce_core.sh" combiner_template.sh \
- > combiner.sh
- chmod 777 combiner.sh
-fi;
+gcc server.c -o exec_server
+gcc client.c -o exec_client
+chmod a+x exec_server
+chmod a+x exec_client
+scp exec_server swift.rcc.uchicago.edu:$MIDWAY_SHARE_FOLDER
+scp exec_client swift.rcc.uchicago.edu:$MIDWAY_SHARE_FOLDER
+scp exec_server login4.beagle.ci.uchicago.edu:$BEAGLE_SHARE_FOLDER
+scp exec_client login4.beagle.ci.uchicago.edu:$BEAGLE_SHARE_FOLDER
+
+
+
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/swiftreduce.swift
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/swiftreduce.swift 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/swiftreduce.swift 2013-10-16 16:14:02 UTC (rev 7167)
@@ -25,10 +25,7 @@
string sites[] = ["beagle", "midway"];
script mapper <"mapper.sh">;
-script uniq <"node_uniq.sh">;
-script combiner <"combiner.sh">;
script reducer <"reducer.sh">;
-script deref <"dereference.sh">;
script aggr <"aggregate.sh">;
fileptr map_out[] <simple_mapper; prefix="map", suffix=".out">;
@@ -47,10 +44,12 @@
app (file out, file err) reduce_bgl (script reducer, string cmd, fileptr all_map){
bgl_bash @reducer cmd @all_map stdout=@out stderr=@err;
}
+
app (file out, file err) reduce_mid (script reducer, string cmd, fileptr all_map){
mid_bash @reducer cmd @all_map stdout=@out stderr=@err;
}
-app (file out, file err) reduce_loc (script reducer, string cmd, file site_results[]){
+
+app (file out, file err) reduce_local (script reducer, string cmd, file site_results[]){
loc_bash @reducer cmd @site_results stdout=@out stderr=@err;
}
@@ -82,5 +81,5 @@
(redsite_out[1], redsite_err[1]) = reduce_mid (reducer, "local", all_map);
file final_out <"final_result">;
file final_err <"final_errors">;
-(final_out, final_err) = reduce_loc (reducer, "naive", redsite_out);
+(final_out, final_err) = reduce_local (reducer, "naive", redsite_out);
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/tc.data
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/tc.data 2013-10-16 15:31:49 UTC (rev 7166)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/tc.data 2013-10-16 16:14:02 UTC (rev 7167)
@@ -1,4 +1,4 @@
-beagle bash /bin/bash null null null
+#beagle bash /bin/bash null null null
midway bash /bin/bash null null null
midway mid_bash /bin/bash null null null
beagle bgl_bash /bin/bash null null null
More information about the Swift-commit
mailing list