[Swift-user] output as an array of files
Allan Espinosa
aespinosa at cs.uchicago.edu
Mon Feb 7 14:31:11 CST 2011
I'm doing this type of workflow to guarantee "that common" input
files in an app() function gets scheduled on a single site resource.
The workflow below does not run the echo() app. With
concurrent_mapper, files[] is undefined so it does not resolve to
anything when invoked by @filenames(). I switched to array_mapper for
explicit output file definitions but it swift looks like it's waiting
for something
type file;
app(file out[]) echo(string s[]) {
echo s @filenames(out) stdout=@filenames(out);
}
app(file out) single(string s[]) {
echo s stdout=@filename(out);
}
file files[] <array_mapper; files=["one.txt", "two.txt", "three.txt"]>;
files = echo(["Hello world", "foo bar"]);
log entry:
2011-02-07 14:27:36,947-0600 INFO unknown RUNID id=run:20110207-1427-stmicsda
2011-02-07 14:27:36,993-0600 INFO CreateArray CREATEARRAY START
array=dataset:20110207-1427-75pv6fs7:720000000013
2011-02-07 14:27:36,994-0600 INFO CreateArray CREATEARRAY START
array=dataset:20110207-1427-75pv6fs7:720000000014
2011-02-07 14:27:36,995-0600 INFO CreateArray CREATEARRAY START
array=dataset:20110207-1427-75pv6fs7:720000000015
2011-02-07 14:27:36,996-0600 INFO CreateArray CREATEARRAY MEMBER
array=dataset:20110207-1427-75pv6fs7:720000000013 index=0
member=dataset:20110207-1427-75pv6fs7:720000000003
2011-02-07 14:27:36,996-0600 INFO CreateArray CREATEARRAY MEMBER
array=dataset:20110207-1427-75pv6fs7:720000000013 index=1
member=dataset:20110207-1427-75pv6fs7:720000000007
2011-02-07 14:27:36,996-0600 INFO CreateArray CREATEARRAY COMPLETED
array=dataset:20110207-1427-75pv6fs7:720000000013
2011-02-07 14:27:36,996-0600 INFO CreateArray CREATEARRAY MEMBER
array=dataset:20110207-1427-75pv6fs7:720000000014 index=0
member=dataset:20110207-1427-75pv6fs7:720000000008
2011-02-07 14:27:36,996-0600 INFO CreateArray CREATEARRAY MEMBER
array=dataset:20110207-1427-75pv6fs7:720000000015 index=0
member=dataset:20110207-1427-75pv6fs7:720000000001
2011-02-07 14:27:36,996-0600 INFO CreateArray CREATEARRAY MEMBER
array=dataset:20110207-1427-75pv6fs7:720000000014 index=1
member=dataset:20110207-1427-75pv6fs7:720000000005
2011-02-07 14:27:36,996-0600 INFO CreateArray CREATEARRAY MEMBER
array=dataset:20110207-1427-75pv6fs7:720000000015 index=1
member=dataset:20110207-1427-75pv6fs7:720000000002
2011-02-07 14:27:36,996-0600 INFO CreateArray CREATEARRAY MEMBER
array=dataset:20110207-1427-75pv6fs7:720000000015 index=2
member=dataset:20110207-1427-75pv6fs7:720000000006
2011-02-07 14:27:36,996-0600 INFO CreateArray CREATEARRAY COMPLETED
array=dataset:20110207-1427-75pv6fs7:720000000015
2011-02-07 14:27:36,997-0600 INFO CreateArray CREATEARRAY MEMBER
array=dataset:20110207-1427-75pv6fs7:720000000014 index=2
member=dataset:20110207-1427-75pv6fs7:720000000004
2011-02-07 14:27:36,997-0600 INFO CreateArray CREATEARRAY COMPLETED
array=dataset:20110207-1427-75pv6fs7:720000000014
2011-02-07 14:27:36,998-0600 DEBUG echo PROCEDURE line=3 thread=0-2 name=echo
jstack trace (no deadlocks found):
2011-02-07 14:28:29
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode):
"Attach Listener" daemon prio=10 tid=0x0000000047afa000 nid=0x1e52
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Progress ticker" daemon prio=10 tid=0x0000000047af8000 nid=0x1e0e
waiting on condition [0x0000000041b11000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.griphyn.vdl.karajan.lib.RuntimeStats$ProgressTicker.run(RuntimeStats.java:137)
Locked ownable synchronizers:
- None
"Restart Log Sync" daemon prio=10 tid=0x0000000047b03800 nid=0x1e0d in
Object.wait() [0x00000000425b5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab601926f8> (a
org.globus.cog.karajan.workflow.nodes.restartLog.SyncThread)
at java.lang.Object.wait(Object.java:485)
at org.globus.cog.karajan.workflow.nodes.restartLog.SyncThread.run(SyncThread.java:45)
- locked <0x00002aab601926f8> (a
org.globus.cog.karajan.workflow.nodes.restartLog.SyncThread)
Locked ownable synchronizers:
- None
"Overloaded Host Monitor" daemon prio=10 tid=0x0000000047b48800
nid=0x1e0c waiting on condition [0x00000000424b4000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.globus.cog.karajan.scheduler.OverloadedHostMonitor.run(OverloadedHostMonitor.java:47)
Locked ownable synchronizers:
- None
"Timer-0" daemon prio=10 tid=0x0000000047b4a800 nid=0x1e0b in
Object.wait() [0x00000000423b3000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab5e500ee8> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aab5e500ee8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
Locked ownable synchronizers:
- None
"NBS0" daemon prio=10 tid=0x0000000047b88800 nid=0x1e0a waiting on
condition [0x00000000422b2000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aab5e4477f8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"pool-1-thread-4" prio=10 tid=0x0000000047a87000 nid=0x1e09 in
Object.wait() [0x00000000421b1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab615a06b0> (a
edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at java.lang.Object.wait(Object.java:485)
at edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:315)
- locked <0x00002aab615a06b0> (a
edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:667)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"pool-1-thread-3" prio=10 tid=0x0000000047902800 nid=0x1e08 in
Object.wait() [0x00000000420b0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab615a06b0> (a
edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at java.lang.Object.wait(Object.java:485)
at edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:315)
- locked <0x00002aab615a06b0> (a
edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:667)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"pool-1-thread-2" prio=10 tid=0x00000000481e0000 nid=0x1e07 in
Object.wait() [0x0000000041faf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab615a06b0> (a
edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at java.lang.Object.wait(Object.java:485)
at edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:315)
- locked <0x00002aab615a06b0> (a
edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:667)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"pool-1-thread-1" prio=10 tid=0x0000000047e6c000 nid=0x1e06 in
Object.wait() [0x0000000040d25000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab615a06b0> (a
edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at java.lang.Object.wait(Object.java:485)
at edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:315)
- locked <0x00002aab615a06b0> (a
edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:667)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
"Low Memory Detector" daemon prio=10 tid=0x000000004775a000 nid=0x1e01
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"CompilerThread1" daemon prio=10 tid=0x0000000047758000 nid=0x1e00
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"CompilerThread0" daemon prio=10 tid=0x0000000047752800 nid=0x1dff
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" daemon prio=10 tid=0x0000000047750000 nid=0x1dfe
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" daemon prio=10 tid=0x000000004772b800 nid=0x1dfd in
Object.wait() [0x000000004129b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab615e8440> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00002aab615e8440> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Locked ownable synchronizers:
- None
"Reference Handler" daemon prio=10 tid=0x0000000047729800 nid=0x1dfc
in Object.wait() [0x0000000041014000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab615a01b0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aab615a01b0> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"main" prio=10 tid=0x00000000476c8000 nid=0x1df6 in Object.wait()
[0x000000004119a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab615e8520> (a
org.griphyn.vdl.karajan.VDL2ExecutionContext)
at java.lang.Object.wait(Object.java:485)
at org.globus.cog.karajan.workflow.ExecutionContext.waitFor(ExecutionContext.java:261)
- locked <0x00002aab615e8520> (a
org.griphyn.vdl.karajan.VDL2ExecutionContext)
at org.griphyn.vdl.karajan.Loader.main(Loader.java:197)
Locked ownable synchronizers:
- None
"VM Thread" prio=10 tid=0x0000000047725000 nid=0x1dfb runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000476db000
nid=0x1df7 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000476dd000
nid=0x1df8 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00000000476de800
nid=0x1df9 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00000000476e0800
nid=0x1dfa runnable
"VM Periodic Task Thread" prio=10 tid=0x0000000047765000 nid=0x1e02
waiting on condition
JNI global references: 1407
--
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>
More information about the Swift-user
mailing list