[Swift-user] Mystery of staging out glut
Lorenzo Pesce
lpesce at uchicago.edu
Mon Dec 17 15:05:39 CST 2012
I am not sure I understand this.
I am running a job which involved a few million tasks over about 10K cores. It seems that when the swift process reaches some memory size (about 10 GB) it starts to produce a lot of stage out glut (right now I have about 5K tasks in stage out), then the number of active tasks drops and eventually I have to kill the job.
Have anyone seen a similar behavior?
This is what the screen trace looks like:
Progress: time: Mon, 17 Dec 2012 20:56:26 +0000 Selecting site:27662 Stage in:4641 Submitting:1 Active:5430 Stage out:4044 Finished in previous run:1253574 Finished successfully:29963
Progress: time: Mon, 17 Dec 2012 20:56:27 +0000 Selecting site:27611 Stage in:4664 Submitting:2 Active:5385 Stage out:4065 Finished in previous run:1253574 Finished successfully:30014
Progress: time: Mon, 17 Dec 2012 20:56:28 +0000 Selecting site:27576 Stage in:4678 Submitting:1 Active:5406 Stage out:4032 Finished in previous run:1253574 Finished successfully:30048
Progress: time: Mon, 17 Dec 2012 20:56:52 +0000 Selecting site:27574 Stage in:4678 Submitted:1 Active:5408 Stage out:4029 Finished in previous run:1253574 Finished successfully:30051
Progress: time: Mon, 17 Dec 2012 20:56:53 +0000 Selecting site:27543 Stage in:4688 Submitting:2 Active:5361 Stage out:4066 Finished in previous run:1253574 Finished successfully:30081
Progress: time: Mon, 17 Dec 2012 20:57:18 +0000 Selecting site:27522 Stage in:4702 Submitted:2 Active:5367 Stage out:4046 Finished in previous run:1253574 Finished successfully:30102
Progress: time: Mon, 17 Dec 2012 20:57:19 +0000 Selecting site:27457 Stage in:4725 Submitted:1 Active:5349 Stage out:4041 Finished in previous run:1253574 Finished successfully:30168
Progress: time: Mon, 17 Dec 2012 20:57:43 +0000 Selecting site:27439 Stage in:4733 Submitting:1 Active:5358 Stage out:4025 Finished in previous run:1253574 Finished successfully:30185
Progress: time: Mon, 17 Dec 2012 20:57:44 +0000 Selecting site:27360 Stage in:4786 Submitting:2 Active:5295 Stage out:4034 Finished in previous run:1253574 Finished successfully:30264
Progress: time: Mon, 17 Dec 2012 20:58:11 +0000 Selecting site:27351 Stage in:4788 Submitting:2 Active:5302 Stage out:4021 Finished in previous run:1253574 Finished successfully:30277
Progress: time: Mon, 17 Dec 2012 20:58:12 +0000 Selecting site:27291 Stage in:4834 Active:5228 Stage out:4054 Finished in previous run:1253574 Finished successfully:30334
Progress: time: Mon, 17 Dec 2012 20:58:37 +0000 Selecting site:27193 Stage in:4895 Active:5263 Checking status:1 Stage out:3958 Finished in previous run:1253574 Finished successfully:30431
Progress: time: Mon, 17 Dec 2012 20:58:38 +0000 Selecting site:27188 Stage in:4895 Submitting:2 Submitted:1 Active:5183 Stage out:4035 Finished in previous run:1253574 Finished successfully:30437
Progress: time: Mon, 17 Dec 2012 20:58:39 +0000 Selecting site:27184 Stage in:4898 Submitted:3 Active:5184 Stage out:4031 Finished in previous run:1253574 Finished successfully:30441
Progress: time: Mon, 17 Dec 2012 20:58:41 +0000 Selecting site:27182 Stage in:4897 Submitting:2 Submitted:1 Active:5187 Stage out:4030 Finished in previous run:1253574 Finished successfully:30442
Progress: time: Mon, 17 Dec 2012 20:58:43 +0000 Selecting site:27182 Stage in:4897 Submitting:2 Active:5188 Stage out:4030 Finished in previous run:1253574 Finished successfully:30442
Progress: time: Mon, 17 Dec 2012 20:58:51 +0000 Selecting site:27182 Stage in:4897 Submitting:1 Submitted:1 Active:5188 Stage out:4030 Finished in previous run:1253574 Finished successfully:30442
Progress: time: Mon, 17 Dec 2012 20:58:52 +0000 Selecting site:27182 Stage in:4897 Submitted:2 Active:5188 Stage out:4029 Finished in previous run:1253574 Finished successfully:30443
Progress: time: Mon, 17 Dec 2012 20:58:53 +0000 Selecting site:27181 Stage in:4897 Submitting:1 Active:5190 Stage out:4029 Finished in previous run:1253574 Finished successfully:30443
Progress: time: Mon, 17 Dec 2012 20:59:01 +0000 Selecting site:27179 Stage in:4898 Submitting:2 Active:5190 Stage out:4027 Finished in previous run:1253574 Finished successfully:30445
Progress: time: Mon, 17 Dec 2012 20:59:26 +0000 Selecting site:27179 Stage in:4898 Submitting:2 Active:5190 Stage out:4027 Finished in previous run:1253574 Finished successfully:30445
Progress: time: Mon, 17 Dec 2012 20:59:27 +0000 Selecting site:27102 Stage in:4952 Submitted:1 Active:5051 Stage out:4113 Finished in previous run:1253574 Finished successfully:30522
Progress: time: Mon, 17 Dec 2012 20:59:52 +0000 Selecting site:27092 Stage in:4955 Submitting:3 Active:5055 Checking status:1 Stage out:4103 Finished in previous run:1253574 Finished successfully:30532
Progress: time: Mon, 17 Dec 2012 20:59:56 +0000 Selecting site:27092 Stage in:4955 Submitting:3 Active:4989 Stage out:4170 Finished in previous run:1253574 Finished successfully:30532
Progress: time: Mon, 17 Dec 2012 20:59:58 +0000 Selecting site:27092 Stage in:4954 Submitting:1 Submitted:2 Active:4990 Stage out:4170 Finished in previous run:1253574 Finished successfully:30532
Progress: time: Mon, 17 Dec 2012 20:59:59 +0000 Selecting site:27091 Stage in:4955 Submitted:1 Active:4991 Checking status:1 Stage out:4169 Finished in previous run:1253574 Finished successfully:30533
Progress: time: Mon, 17 Dec 2012 21:00:01 +0000 Selecting site:27091 Stage in:4955 Active:4991 Stage out:4171 Finished in previous run:1253574 Finished successfully:30533
Progress: time: Mon, 17 Dec 2012 21:00:02 +0000 Selecting site:27091 Stage in:4955 Active:4990 Checking status:1 Stage out:4171 Finished in previous run:1253574 Finished successfully:30533
Progress: time: Mon, 17 Dec 2012 21:00:07 +0000 Selecting site:27091 Stage in:4955 Active:4990 Stage out:4171 Finished in previous run:1253574 Finished successfully:30534
Progress: time: Mon, 17 Dec 2012 21:00:09 +0000 Selecting site:27089 Stage in:4957 Active:4990 Stage out:4169 Finished in previous run:1253574 Finished successfully:30536
Progress: time: Mon, 17 Dec 2012 21:00:10 +0000 Selecting site:27087 Stage in:4958 Submitting:1 Active:4990 Stage out:4167 Finished in previous run:1253574 Finished successfully:30538
Progress: time: Mon, 17 Dec 2012 21:00:11 +0000 Selecting site:27085 Stage in:4960 Submitted:1 Active:4990 Stage out:4166 Finished in previous run:1253574 Finished successfully:30539
Progress: time: Mon, 17 Dec 2012 21:00:37 +0000 Selecting site:27065 Stage in:4967 Submitting:1 Active:4949 Stage out:4198 Finished in previous run:1253574 Finished successfully:30561
Progress: time: Mon, 17 Dec 2012 21:00:38 +0000 Selecting site:27018 Stage in:5000 Submitting:1 Active:4810 Stage out:4305 Finished in previous run:1253574 Finished successfully:30607
Progress: time: Mon, 17 Dec 2012 21:01:03 +0000 Selecting site:26988 Stage in:5003 Submitting:6 Active:4828 Stage out:4279 Finished in previous run:1253574 Finished successfully:30637
Progress: time: Mon, 17 Dec 2012 21:01:04 +0000 Selecting site:26963 Stage in:5016 Active:4726 Checking status:1 Stage out:4374 Finished in previous run:1253574 Finished successfully:30661
Progress: time: Mon, 17 Dec 2012 21:01:29 +0000 Selecting site:26930 Stage in:5037 Submitted:4 Active:4724 Stage out:4352 Finished in previous run:1253574 Finished successfully:30694
Progress: time: Mon, 17 Dec 2012 21:01:30 +0000 Selecting site:26864 Stage in:5083 Active:4649 Stage out:4385 Finished in previous run:1253574 Finished successfully:30760
Progress: time: Mon, 17 Dec 2012 21:01:55 +0000 Selecting site:26858 Stage in:5085 Submitting:1 Active:4651 Stage out:4380 Finished in previous run:1253574 Finished successfully:30766
Progress: time: Mon, 17 Dec 2012 21:01:56 +0000 Selecting site:26811 Stage in:5119 Submitted:1 Active:4561 Stage out:4436 Finished in previous run:1253574 Finished successfully:30813
Progress: time: Mon, 17 Dec 2012 21:02:21 +0000 Selecting site:26805 Stage in:5124 Active:4562 Stage out:4431 Finished in previous run:1253574 Finished successfully:30819
Progress: time: Mon, 17 Dec 2012 21:02:22 +0000 Selecting site:26745 Stage in:5157 Active:4487 Stage out:4473 Finished in previous run:1253574 Finished successfully:30879
Progress: time: Mon, 17 Dec 2012 21:02:47 +0000 Selecting site:26704 Stage in:5165 Submitted:3 Active:4514 Stage out:4435 Finished in previous run:1253574 Finished successfully:30920
Progress: time: Mon, 17 Dec 2012 21:02:48 +0000 Selecting site:26671 Stage in:5171 Submitted:1 Active:4450 Stage out:4495 Finished in previous run:1253574 Finished successfully:30953
Progress: time: Mon, 17 Dec 2012 21:03:13 +0000 Selecting site:26667 Stage in:5172 Submitting:1 Active:4451 Stage out:4494 Finished in previous run:1253574 Finished successfully:30956
Progress: time: Mon, 17 Dec 2012 21:03:38 +0000 Selecting site:26603 Stage in:5197 Submitting:2 Active:4380 Stage out:4539 Finished in previous run:1253574 Finished successfully:31020
Progress: time: Mon, 17 Dec 2012 21:03:39 +0000 Selecting site:26562 Stage in:5220 Submitted:2 Active:4305 Stage out:4591 Finished in previous run:1253574 Finished successfully:31061
Progress: time: Mon, 17 Dec 2012 21:04:04 +0000 Selecting site:26559 Stage in:5222 Submitted:1 Active:4307 Stage out:4588 Finished in previous run:1253574 Finished successfully:31064
Progress: time: Mon, 17 Dec 2012 21:04:05 +0000 Selecting site:26509 Stage in:5232 Submitting:3 Active:4230 Stage out:4653 Finished in previous run:1253574 Finished successfully:31114
Progress: time: Mon, 17 Dec 2012 21:04:30 +0000 Selecting site:26470 Stage in:5245 Active:4253 Checking status:1 Stage out:4619 Finished in previous run:1253574 Finished successfully:31153
Progress: time: Mon, 17 Dec 2012 21:04:31 +0000 Selecting site:26425 Stage in:5274 Submitting:3 Active:4158 Stage out:4683 Finished in previous run:1253574 Finished successfully:31198
Progress: time: Mon, 17 Dec 2012 21:04:55 +0000 Selecting site:26422 Stage in:5277 Active:4161 Stage out:4680 Finished in previous run:1253574 Finished successfully:31201
More information about the Swift-user
mailing list