[Nek5000-users] Running NEK on CRAY; compilation & running issues

Mon Jan 30 04:23:40 CST 2012

Dear NEKs,

I'm currently trying to compile NEK via crayftn to run on a XE6 and I 
run into some issues. Some compilation problems I solved (with kind help 
of CRAY personnel). I'll mention them here as they can be part of the 
problem, I assume.

1) changed
*ftn*) P="-r8 -Mpreprocess
to *ftn*) P="-s real64 -eZ -em"
for double precision reals and to invoke preprocessor in makenek.inc 
(Cray uses ftn command as a wrapper)

2) changed subroutine name “gsync” to “gsync_nek” in all source files as 
crayftn has a build-in routine of the same name that conflicts with 
Nek's gsync

3) changed calls to subroutine “multd” from
CALL MULTD (TA1,TVX,RXM2,SXM2,TXM2,1)
to CALL MULTD (TA1,TVX,RXM2,SXM2,TXM2,1,0)
in “navier1.f” as crayftn complains about missing seventh argument 
“iflg” (calls come from subroutine “tmultd” which seems not to be called 
in the code, though)

In the SIZE file I'm setting lelt=300, lelv=lelt, lp = 64, lelg = 300.
With these changes I'm running the eddy_uv example via “aprun -n 64 -N 
32 ./nek5000” on 64 processors and I'm getting the following output:

/----------------------------------------------------------\\
| _ __ ______ __ __ ______ ____ ____ ____ |
| / | / // ____// //_/ / ____/ / __ \\ / __ \\ / __ \\ |
| / |/ // __/ / ,< /___ \\ / / / // / / // / / / |
| / /| // /___ / /| | ____/ / / /_/ // /_/ // /_/ / |
| /_/ |_//_____//_/ |_|/_____/ \\____/ \\____/ \\____/ |
| |
|----------------------------------------------------------|
| |
| NEK5000: Open Source Spectral Element Solver |
| COPYRIGHT (c) 2008-2010 UCHICAGO ARGONNE, LLC |
| Version: 1.0rc1 / SVN r730 |
| Web: http://nek5000.mcs.anl.gov |
| |
\\----------------------------------------------------------/

Number of processors: 64
REAL wdsize : 8
INTEGER wdsize : 4

Beginning session:
/zhome/academic/HLRS/iag/iagoschm/run_nek/eddy_example/eddy_uv.rea

timer accuracy: 2.8610229E-07 sec

read .rea file
nelgt/nelgv/lelt: 256 256 300
lx1 /lx2 /lx3 : 8 6 8

mapping elements to processors
0, 2*4, 2*256 NELV
1, 2*4, 2*256 NELV
8, 2*4, 2*256 NELV
9, 2*4, 2*256 NELV
17, 2*4, 2*256 NELV
16, 2*4, 2*256 NELV
20, 2*4, 2*256 NELV
3*4, 2*256 NELV
29, 2*4, 2*256 NELV
21, 2*4, 2*256 NELV
28, 2*4, 2*256 NELV
7, 2*4, 2*256 NELV
31, 2*4, 2*256 NELV
25, 2*4, 2*256 NELV
11, 2*4, 2*256 NELV
12, 2*4, 2*256 NELV
6, 2*4, 2*256 NELV
30, 2*4, 2*256 NELV
18, 2*4, 2*256 NELV
19, 2*4, 2*256 NELV
2, 2*4, 2*256 NELV
5, 2*4, 2*256 NELV
23, 2*4, 2*256 NELV
27, 2*4, 2*256 NELV
10, 2*4, 2*256 NELV
22, 2*4, 2*256 NELV
13, 2*4, 2*256 NELV
14, 2*4, 2*256 NELV
15, 2*4, 2*256 NELV
3, 2*4, 2*256 NELV
26, 2*4, 2*256 NELV
24, 2*4, 2*256 NELV
33, 2*4, 2*256 NELV
41, 2*4, 2*256 NELV
32, 2*4, 2*256 NELV
40, 2*4, 2*256 NELV
45, 2*4, 2*256 NELV
44, 2*4, 2*256 NELV
49, 2*4, 2*256 NELV
34, 2*4, 2*256 NELV
35, 2*4, 2*256 NELV
48, 2*4, 2*256 NELV
39, 2*4, 2*256 NELV
38, 2*4, 2*256 NELV
50, 2*4, 2*256 NELV
51, 2*4, 2*256 NELV
37, 2*4, 2*256 NELV
36, 2*4, 2*256 NELV
55, 2*4, 2*256 NELV
54, 2*4, 2*256 NELV
57, 2*4, 2*256 NELV
58, 2*4, 2*256 NELV
43, 2*4, 2*256 NELV
42, 2*4, 2*256 NELV
47, 2*4, 2*256 NELV
53, 2*4, 2*256 NELV
59, 2*4, 2*256 NELV
46, 2*4, 2*256 NELV
60, 2*4, 2*256 NELV
61, 2*4, 2*256 NELV
63, 2*4, 2*256 NELV
62, 2*4, 2*256 NELV
52, 2*4, 2*256 NELV
56, 2*4, 2*256 NELV
25, 0, 2*4, 256 NELT FAIL
24, 0, 2*4, 256 NELT FAIL
21, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
20, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
31, 0, 2*4, 256 NELT FAIL
8, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
17, 0, 2*4, 256 NELT FAIL
14, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
15, 0, 2*4, 256 NELT FAIL
16, 0, 2*4, 256 NELT FAIL
26, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
Check that .map file and .rea file agree
10, 0, 2*4, 256 NELT FAIL
2, 0, 2*4, 256 NELT FAIL
30, 0, 2*4, 256 NELT FAIL
11, 0, 2*4, 256 NELT FAIL
7, 0, 2*4, 256 NELT FAIL
1, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
2*0, 2*4, 256 NELT FAIL
6, 0, 2*4, 256 NELT FAIL
4, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
13, 0, 2*4, 256 NELT FAIL
12, 0, 2*4, 256 NELT FAIL
28, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
18, 0, 2*4, 256 NELT FAIL
29, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
23, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
27, 0, 2*4, 256 NELT FAIL
3, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
19, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
5, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
22, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
Check that .map file and .rea file agree
2*0, 2*4, 256 NELT FB
1, 0, 2*4, 256 NELT FB
2, 0, 2*4, 256 NELT FB
3, 0, 2*4, 256 NELT FB
4, 0, 2*4, 256 NELT FB
5, 0, 2*4, 256 NELT FB
6, 0, 2*4, 256 NELT FB
7, 0, 2*4, 256 NELT FB
8, 0, 2*4, 256 NELT FB
9, 3*4, 256 NELT FB
10, 0, 2*4, 256 NELT FB
11, 0, 2*4, 256 NELT FB
12, 0, 2*4, 256 NELT FB
13, 0, 2*4, 256 NELT FB
14, 0, 2*4, 256 NELT FB
15, 0, 2*4, 256 NELT FB
16, 0, 2*4, 256 NELT FB
17, 0, 2*4, 256 NELT FB
18, 0, 2*4, 256 NELT FB
19, 0, 2*4, 256 NELT FB
20, 0, 2*4, 256 NELT FB
21, 0, 2*4, 256 NELT FB
22, 0, 2*4, 256 NELT FB
23, 0, 2*4, 256 NELT FB
24, 0, 2*4, 256 NELT FB
25, 0, 2*4, 256 NELT FB
26, 0, 2*4, 256 NELT FB
27, 0, 2*4, 256 NELT FB
28, 0, 2*4, 256 NELT FB
29, 0, 2*4, 256 NELT FB
30, 0, 2*4, 256 NELT FB
31, 0, 2*4, 256 NELT FB

call exitt: dying ...

backtrace(): obtained 1 stack frames.
[0x662a40]

total elapsed time : 7.82199E-02 sec
total solver time incl. I/O : 0.00000E+00 sec
time/timestep : 0.00000E+00 sec
CPU seconds/timestep/gridpt : 0.00000E+00 sec

39, 0, 2*4, 256 NELT FAIL
38, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
Check that .map file and .rea file agree
36, 0, 2*4, 256 NELT FAIL
41, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
32, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
48, 0, 2*4, 256 NELT FAIL
34, 0, 2*4, 256 NELT FAIL
54, 196, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
40, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
Check that .map file and .rea file agree
46, 0, 2*4, 256 NELT FAIL
47, 0, 2*4, 256 NELT FAIL
33, 0, 2*4, 256 NELT FAIL
37, 0, 2*4, 256 NELT FAIL
35, 0, 2*4, 256 NELT FAIL
43, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
45, 0, 2*4, 256 NELT FAIL
42, 0, 2*4, 256 NELT FAIL
44, 0, 2*4, 256 NELT FAIL
Check that .map file and .rea file agree
Check that .map file and .rea file agree
Check that .map file and .rea file agree
32, 0, 2*4, 256 NELT FB
33, 0, 2*4, 256 NELT FB
34, 0, 2*4, 256 NELT FB
35, 0, 2*4, 256 NELT FB
36, 0, 2*4, 256 NELT FB
37, 0, 2*4, 256 NELT FB
38, 0, 2*4, 256 NELT FB
39, 0, 2*4, 256 NELT FB
40, 0, 2*4, 256 NELT FB
41, 0, 2*4, 256 NELT FB
42, 0, 2*4, 256 NELT FB
43, 0, 2*4, 256 NELT FB
44, 0, 2*4, 256 NELT FB
45, 0, 2*4, 256 NELT FB
46, 0, 2*4, 256 NELT FB
47, 0, 2*4, 256 NELT FB
48, 0, 2*4, 256 NELT FB
49, 3*4, 256 NELT FB
50, 3*4, 256 NELT FB
51, 3*4, 256 NELT FB
52, 3*4, 256 NELT FB
53, 3*4, 256 NELT FB
54, 196, 2*4, 256 NELT FB
55, 3*4, 256 NELT FB
56, 3*4, 256 NELT FB
57, 3*4, 256 NELT FB
58, 3*4, 256 NELT FB
59, 3*4, 256 NELT FB
60, 3*4, 256 NELT FB
61, 3*4, 256 NELT FB
62, 3*4, 256 NELT FB
63, 3*4, 256 NELT FB
Application 419539 resources: utime ~8s, stime ~13s

Does the code map in parallel here? I'm not sure what's happening here, 
because the .map and .rea file do in fact agree. The error seems to 
occur when processor #25 is mapped for a second time. The same thing 
happens with my own cases that were running on a similar machine before 
but with Nek compiled with PGI.

Any help would be greatly appreciated

Oliver