[Swift-commit] r6007 - SwiftApps/ParVis/HiRAMTools
tmoore at ci.uchicago.edu
tmoore at ci.uchicago.edu
Tue Nov 6 15:17:43 CST 2012
Author: tmoore
Date: 2012-11-06 15:17:42 -0600 (Tue, 06 Nov 2012)
New Revision: 6007
Added:
SwiftApps/ParVis/HiRAMTools/makeyearly2yr.swift
SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh
Modified:
SwiftApps/ParVis/HiRAMTools/README
SwiftApps/ParVis/HiRAMTools/combine_realization.sh
SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh
Log:
updated HiRAMTools scripts to handle 2year configurations
Modified: SwiftApps/ParVis/HiRAMTools/README
===================================================================
--- SwiftApps/ParVis/HiRAMTools/README 2012-11-05 03:44:51 UTC (rev 6006)
+++ SwiftApps/ParVis/HiRAMTools/README 2012-11-06 21:17:42 UTC (rev 6007)
@@ -231,3 +231,17 @@
- Naming conventions for outdir now used in combine need to be added to
makeyearly
+
+
+FILE NAMING CONVENTIONS:
+
+- under the combine data sets, we'll use a manual soft link to the successfully
+ completed realization called "done". This is a manual setting to identify
+ which subdirectory is fully complete.
+
+- under annualize, the annualize script should point to the "done" directory
+ within a realization combine directory, but not recreate the "done" directory.
+ A "done" soft link will be created for the completed annualize directory
+ *current swift script does not yet conform to the above.
+
+
Modified: SwiftApps/ParVis/HiRAMTools/combine_realization.sh
===================================================================
--- SwiftApps/ParVis/HiRAMTools/combine_realization.sh 2012-11-05 03:44:51 UTC (rev 6006)
+++ SwiftApps/ParVis/HiRAMTools/combine_realization.sh 2012-11-06 21:17:42 UTC (rev 6007)
@@ -60,8 +60,7 @@
<execution provider="coaster" jobmanager="local:cobalt" url="eureka.alcf.anl.gov" />
<filesystem provider="local" />
<!-- <profile namespace="globus"key="internalHostname">172.17.3.11</profile> -->
- <!-- <profile namespace="globus" key="project">SU_Climate</profile> -->
- <profile namespace="globus" key="project">MTCScienceApps</profile>
+ <profile namespace="globus" key="project">SU_Climate</profile>
<profile namespace="globus" key="queue">default</profile>
<profile namespace="karajan" key="jobthrottle">2.56</profile>
<profile namespace="karajan" key="initialScore">10000</profile>
Added: SwiftApps/ParVis/HiRAMTools/makeyearly2yr.swift
===================================================================
--- SwiftApps/ParVis/HiRAMTools/makeyearly2yr.swift (rev 0)
+++ SwiftApps/ParVis/HiRAMTools/makeyearly2yr.swift 2012-11-06 21:17:42 UTC (rev 6007)
@@ -0,0 +1,24 @@
+
+type file;
+
+type ncrec {
+ string f1;
+}
+
+string rundir=@arg("rundir","NO-OUTDIR-SPECIFIED");
+string ncfile=@arg("ncfile","ncfiles");
+
+app (file sout, file serr) makeyearly (string dir, string f1)
+{
+ makeyearly_2yr f1 dir stdout=@sout stderr=@serr;
+}
+
+ncrec nc[] = readData(ncfile);
+
+foreach n,i in nc {
+ file so <single_file_mapper; file=@strcat("stdlog/stdout.",i)>;
+ file se <single_file_mapper; file=@strcat("stdlog/stderr.",i)>;
+ (so,se) = makeyearly(rundir,n.f1);
+}
+
+
Added: SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh
===================================================================
--- SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh (rev 0)
+++ SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh 2012-11-06 21:17:42 UTC (rev 6007)
@@ -0,0 +1,88 @@
+#! /bin/bash
+
+# Create a fast (local-filesystem) working dir
+
+if mkdir -p /scratch/wilde; then
+ scratch=/scratch/wilde
+elif mkdir -p $HOME/scratch; then
+ scratch=$HOME/scratch
+else
+ echo $0: cant create scratch dir
+ exit 11
+fi
+
+tmp=$(mktemp -d $scratch/makeyearly_XXXXXX)
+startdir=$PWD
+cd $tmp
+
+echo running in $PWD
+
+# the following is not needed for a two-year case
+#qtrdec1=$1 # 1st quarter-decade dir
+#qtrdec2=$2 # 2nd quarter-decade dir
+
+the2year=$1
+destbase=$2 # base dir for output years
+
+echo destbase=$destbase
+
+# Add cdo and nco tools to PATH and Intel compiler v11 libs to LD_LIBRARY_PATH
+
+#export PATH=/soft/apps/cdo-1.4.7/bin:/soft/apps/nco-4.0.9/bin:$PATH
+export PATH=/soft/apps/nco-4.0.9/bin:$PATH
+export LD_LIBRARY_PATH=/soft/apps/intel-fc-11.0.074/lib/intel64:/soft/apps/intel-cc-11.0.074/lib/intel64:/soft/apps/netcdf-3.6.3/lib:/usr/lib:/usr/X11R6/lib:/usr/local/lib
+
+fname=$(basename $the2year .nc)
+firstyear=$(basename $(dirname $the2year) | sed -e 's/....$//')
+
+# process only the time-based files
+echo $fname
+if echo $fname | egrep -v '.nc'; then
+
+ # Merge 2 quarter-decade files into one half-decade file
+
+# ncrcat $qtrdec1 $qtrdec2 5years.nc
+
+ # Extract each year from the half-decade file
+echo "processing ${fname}"
+ for (( y=$firstyear; y<$firstyear+2; y++ )); do
+ ncks -d time,"${y}-01-01 00:00:00","${y}-12-31 23:59:59" $the2year $y.nc
+ done
+
+fi
+
+# Copy the years from temporary to permanent filesystem
+
+for (( y=$firstyear; y<$firstyear+2; y++ ))
+
+do
+ destdir=$destbase/${y}0101
+ mkdir -p $destdir
+ if echo $fname | egrep -v 'grid_spec|scalar'; then
+ dd if=$y.nc of=$destdir/$fname.nc bs=8M
+ else
+ dd if=$the2year of=$destdir/$fname.nc bs=8M
+ fi
+done
+
+rm -rf $tmp
+
+# Delete when this works:
+
+ # Old method:
+ #
+ # cdo mergetime $qtrdec1 $qtrdec2 ${fname}.nc
+ # cdo splityear ${fname}.nc ${fname}.
+ # Yields, eg, from atmos_month.tile1:
+ # atmos_month.tile1.1950.nc
+ # atmos_month.tile1.1951.nc
+ # atmos_month.tile1.1952.nc
+ # atmos_month.tile1.1953.nc
+ # atmos_month.tile1.1954.nc
+
+ # New method:
+ #
+ # ncrcat input1.nc input2.nc 5years.nc
+ # ncks -d time,"$yyyy-01-01 00:00:00","$yyyy-12-31 23:59:59" 5years.nc year1.nc
+
+
Property changes on: SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh
___________________________________________________________________
Added: svn:executable
+ *
Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh
===================================================================
--- SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh 2012-11-05 03:44:51 UTC (rev 6006)
+++ SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh 2012-11-06 21:17:42 UTC (rev 6007)
@@ -16,6 +16,12 @@
# Default assumes cdir is a rundir from combine script
# New rundir will be appended to end of outsuffix
+# Strip out the "done" directory in the outsuffix if present
+
+STRIPTEXT="\/done"
+
+outsuffix=${outsuffix//$STRIPTEXT/}
+
# generate the list of files to process: do every half-decade
echo f1 f2 >ncfiles
More information about the Swift-commit
mailing list