Yuheng,<br><br>I am unable to reproduce your problem, here are the compile commands and the resulting output...<br><br>jedwards@fr0101ge:~/src/pnetcdf/test/fandc> make<br>blrts_xlf -o pnctestf -g -qfullpath -O2 -qsuffix=cpp=F -I/bgl/BlueLight/ppcfloor/bglsys/include -qsuffix=cpp=F -I/bgl/BlueLight/ppcfloor/bglsys/include ./pnctestf.F -I../../src/libf/ -L/contrib/bgl/pnetcdf/lib -lpnetcdf -lm -L/bgl/BlueLight/ppcfloor/bglsys/lib -
lmpich.rts -lmsglayer.rts -lrts.rts -ldevices.rts<br>** pnf_test === End of Compilation 1 ===<br>1501-510 Compilation successful for file pnctestf.F.<br>blrts_xlc -o pnctest -g -qfullpath -O2 -I/bgl/BlueLight/ppcfloor/bglsys/include -o pnctest ./pnctest.c -I./../../src/lib -L/contrib/bgl/pnetcdf/lib -lpnetcdf -lm -L/bgl/BlueLight/ppcfloor/bglsys/lib -
lmpich.rts -lmsglayer.rts -lrts.rts -ldevices.rts<br><br><br>mype pe_coords totsiz_3d locsiz_3d kstart,jstart,istart<br> 0 0 0 0 256 256 256 256 256 2 1 1 1<br> 42 0 0 42 256 256 256 256 256 2 85 1 1
<br> 6 0 0 6 256 256 256 256 256 2 13 1 1<br> 72 0 0 72 256 256 256 256 256 2 145 1 1<br> 76 0 0 76 256 256 256 256 256 2 153 1 1
<br> 8 0 0 8 256 256 256 256 256 2 17 1 1<br> 12 0 0 12 256 256 256 256 256 2 25 1 1<br> 64 0 0 64 256 256 256 256 256 2 129 1 1
<br> 68 0 0 68 256 256 256 256 256 2 137 1 1<br> 33 0 0 33 256 256 256 256 256 2 67 1 1<br> 4 0 0 4 256 256 256 256 256 2 9 1 1
<br> 75 0 0 75 256 256 256 256 256 2 151 1 1<br>102 0 0 ** 256 256 256 256 256 2 205 1 1<br> 1 0 0 1 256 256 256 256 256 2 3 1 1
<br> 13 0 0 13 256 256 256 256 256 2 27 1 1<br>105 0 0 ** 256 256 256 256 256 2 211 1 1<br> 77 0 0 77 256 256 256 256 256 2 155 1 1
<br> 34 0 0 34 256 256 256 256 256 2 69 1 1<br> 5 0 0 5 256 256 256 256 256 2 11 1 1<br> 65 0 0 65 256 256 256 256 256 2 131 1 1
<br> 69 0 0 69 256 256 256 256 256 2 139 1 1<br> 40 0 0 40 256 256 256 256 256 2 81 1 1<br> 38 0 0 38 256 256 256 256 256 2 77 1 1
<br> 74 0 0 74 256 256 256 256 256 2 149 1 1<br>103 0 0 ** 256 256 256 256 256 2 207 1 1<br> 10 0 0 10 256 256 256 256 256 2 21 1 1
<br> 45 0 0 45 256 256 256 256 256 2 91 1 1<br>104 0 0 ** 256 256 256 256 256 2 209 1 1<br>109 0 0 ** 256 256 256 256 256 2 219 1 1
<br> 9 0 0 9 256 256 256 256 256 2 19 1 1<br> 31 0 0 31 256 256 256 256 256 2 63 1 1<br> 98 0 0 98 256 256 256 256 256 2 197 1 1
<br> 78 0 0 78 256 256 256 256 256 2 157 1 1<br> 43 0 0 43 256 256 256 256 256 2 87 1 1<br> 44 0 0 44 256 256 256 256 256 2 89 1 1
<br> 73 0 0 73 256 256 256 256 256 2 147 1 1<br>100 0 0 ** 256 256 256 256 256 2 201 1 1<br> 41 0 0 41 256 256 256 256 256 2 83 1 1
<br> 46 0 0 46 256 256 256 256 256 2 93 1 1<br> 67 0 0 67 256 256 256 256 256 2 135 1 1<br> 71 0 0 71 256 256 256 256 256 2 143 1 1
<br> 2 0 0 2 256 256 256 256 256 2 5 1 1<br> 37 0 0 37 256 256 256 256 256 2 75 1 1<br> 97 0 0 97 256 256 256 256 256 2 195 1 1
<br>101 0 0 ** 256 256 256 256 256 2 203 1 1<br> 32 0 0 32 256 256 256 256 256 2 65 1 1<br> 15 0 0 15 256 256 256 256 256 2 31 1 1
<br>106 0 0 ** 256 256 256 256 256 2 213 1 1<br>110 0 0 ** 256 256 256 256 256 2 221 1 1<br> 3 0 0 3 256 256 256 256 256 2 7 1 1
<br> 36 0 0 36 256 256 256 256 256 2 73 1 1<br> 96 0 0 96 256 256 256 256 256 2 193 1 1<br>108 0 0 ** 256 256 256 256 256 2 217 1 1
<br> 11 0 0 11 256 256 256 256 256 2 23 1 1<br> 14 0 0 14 256 256 256 256 256 2 29 1 1<br>114 0 0 ** 256 256 256 256 256 2 229 1 1
<br> 86 0 0 86 256 256 256 256 256 2 173 1 1<br> 49 0 0 49 256 256 256 256 256 2 99 1 1<br> 7 0 0 7 256 256 256 256 256 2 15 1 1
<br> 66 0 0 66 256 256 256 256 256 2 133 1 1<br> 70 0 0 70 256 256 256 256 256 2 141 1 1<br> 35 0 0 35 256 256 256 256 256 2 71 1 1
<br> 54 0 0 54 256 256 256 256 256 2 109 1 1<br> 99 0 0 99 256 256 256 256 256 2 199 1 1<br> 79 0 0 79 256 256 256 256 256 2 159 1 1
<br> 27 0 0 27 256 256 256 256 256 2 55 1 1<br> 47 0 0 47 256 256 256 256 256 2 95 1 1<br>113 0 0 ** 256 256 256 256 256 2 227 1 1
<br>118 0 0 ** 256 256 256 256 256 2 237 1 1<br> 50 0 0 50 256 256 256 256 256 2 101 1 1<br> 61 0 0 61 256 256 256 256 256 2 123 1 1
<br>107 0 0 ** 256 256 256 256 256 2 215 1 1<br>111 0 0 ** 256 256 256 256 256 2 223 1 1<br> 17 0 0 17 256 256 256 256 256 2 35 1 1
<br> 39 0 0 39 256 256 256 256 256 2 79 1 1<br> 83 0 0 83 256 256 256 256 256 2 167 1 1<br>125 0 0 ** 256 256 256 256 256 2 251 1 1
<br> 18 0 0 18 256 256 256 256 256 2 37 1 1<br> 22 0 0 22 256 256 256 256 256 2 45 1 1<br> 81 0 0 81 256 256 256 256 256 2 163 1 1
<br>126 0 0 ** 256 256 256 256 256 2 253 1 1<br> 56 0 0 56 256 256 256 256 256 2 113 1 1<br> 21 0 0 21 256 256 256 256 256 2 43 1 1
<br> 82 0 0 82 256 256 256 256 256 2 165 1 1<br> 85 0 0 85 256 256 256 256 256 2 171 1 1<br> 58 0 0 58 256 256 256 256 256 2 117 1 1
<br> 62 0 0 62 256 256 256 256 256 2 125 1 1<br>120 0 0 ** 256 256 256 256 256 2 241 1 1<br>127 0 0 ** 256 256 256 256 256 2 255 1 1
<br> 24 0 0 24 256 256 256 256 256 2 49 1 1<br> 52 0 0 52 256 256 256 256 256 2 105 1 1<br>122 0 0 ** 256 256 256 256 256 2 245 1 1
<br>124 0 0 ** 256 256 256 256 256 2 249 1 1<br> 51 0 0 51 256 256 256 256 256 2 103 1 1<br> 55 0 0 55 256 256 256 256 256 2 111 1 1
<br> 80 0 0 80 256 256 256 256 256 2 161 1 1<br> 94 0 0 94 256 256 256 256 256 2 189 1 1<br> 48 0 0 48 256 256 256 256 256 2 97 1 1
<br> 28 0 0 28 256 256 256 256 256 2 57 1 1<br> 91 0 0 91 256 256 256 256 256 2 183 1 1<br> 92 0 0 92 256 256 256 256 256 2 185 1 1
<br> 26 0 0 26 256 256 256 256 256 2 53 1 1<br> 30 0 0 30 256 256 256 256 256 2 61 1 1<br>112 0 0 ** 256 256 256 256 256 2 225 1 1
<br> 87 0 0 87 256 256 256 256 256 2 175 1 1<br> 25 0 0 25 256 256 256 256 256 2 51 1 1<br> 60 0 0 60 256 256 256 256 256 2 121 1 1
<br> 90 0 0 90 256 256 256 256 256 2 181 1 1<br>116 0 0 ** 256 256 256 256 256 2 233 1 1<br> 19 0 0 19 256 256 256 256 256 2 39 1 1
<br> 23 0 0 23 256 256 256 256 256 2 47 1 1<br> 89 0 0 89 256 256 256 256 256 2 179 1 1<br> 95 0 0 95 256 256 256 256 256 2 191 1 1
<br> 57 0 0 57 256 256 256 256 256 2 115 1 1<br> 29 0 0 29 256 256 256 256 256 2 59 1 1<br>123 0 0 ** 256 256 256 256 256 2 247 1 1
<br> 93 0 0 93 256 256 256 256 256 2 187 1 1<br> 59 0 0 59 256 256 256 256 256 2 119 1 1<br> 53 0 0 53 256 256 256 256 256 2 107 1 1
<br>121 0 0 ** 256 256 256 256 256 2 243 1 1<br>117 0 0 ** 256 256 256 256 256 2 235 1 1<br> 16 0 0 16 256 256 256 256 256 2 33 1 1
<br> 63 0 0 63 256 256 256 256 256 2 127 1 1<br>115 0 0 ** 256 256 256 256 256 2 231 1 1<br>119 0 0 ** 256 256 256 256 256 2 239 1 1
<br> 20 0 0 20 256 256 256 256 256 2 41 1 1<br> 88 0 0 88 256 256 256 256 256 2 177 1 1<br> 84 0 0 84 256 256 256 256 256 2 169 1 1
<br>write 1: .149E-01 .149E+01<br>write 2: .954E+00 .131E+01<br>write 3: .240E+00 .149E+01<br>write 4: .227E+00 .142E+01<br>write 5: .205E+00 .134E+01<br> read 1: .132E-01 .621E+00<br>diff, delmax, delmin = .000E+00 .000E+00 .000E+00
<br> read 2: .178E-01 .624E+00<br> read 3: .195E-01 .629E+00<br> read 4: .184E-01 .638E+00<br> read 5: .210E-01 .659E+00<br>File size: .671E+02 MB<br> Write: 51.398 MB/s (eff., 50.819 MB/s)<br> Read :
108.142 MB/s (eff., 105.886 MB/s)<br>Total number PEs: 128<br> .149E-01 .131E+01 50.819 .132E-01 .621E+00 105.886<br><br><br><br><div><span class="gmail_quote">On 4/13/06, <b class="gmail_sendername">Yu-Heng Tseng
</b> <<a href="mailto:YHTseng@lbl.gov">YHTseng@lbl.gov</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi Ross<br><br>
I am testing it on NCAR's BG/L. These questions are really beyond my<br>understanding. Sidd at NCAR should be able to answer your questions.<br>Thanks!<br><br>Yuheng<br>---------------------------------------------------<br>
Yu-Heng Tseng<br><br>Computational Research Division<br>Lawrence Berkeley National Laboratory<br>One Cyclotron Rd, MS: 50F-1650<br>Berkeley, CA94720<br><a href="mailto:YHTseng@lbl.gov">YHTseng@lbl.gov</a><br>510.495.2904<br>
<br>----- Original Message -----<br>From: Rob Ross <<a href="mailto:rross@mcs.anl.gov">rross@mcs.anl.gov</a>><br>Date: Thursday, April 13, 2006 6:23 am<br>Subject: Re: Inconsistent results on bluegene<br><br>> Hi,
<br>><br>> I see. Can you tell us what BG/L you are running on, what file<br>> system it<br>> has, and how it is mounted to the I/O nodes?<br>><br>> If not, can you CC someone that would know?<br>><br>
> This could be an NFS thing.<br>><br>> Thanks,<br>><br>> Rob<br>><br>> Yu-Heng Tseng wrote:<br>> > Hi,<br>> ><br>> > Here is the description of the problem.<br>> > In the test directory under test/fandc/
<br>> > there is a Fortran test file called "pnf_test.F"<br>> > This file generates an array has size 256x256x256 and output it<br>> to a<br>> > pnf_test.nc file and read it back.<br>> >
<br>> > In other machines or using gcc to compile the library, it<br>> compares the<br>> > original array with the readin new array. The output have<br>> > diff, delmax, delmin<br>> ><br>> > They should be all zero which represents the difference between
<br>> the<br>> > new array and old array. Unfortunitely, I got some non-zero<br>> values<br>> > when I use 2, 8, 16 processors which means the new readin array<br>> and<br>> > original array are not identical.
<br>> > The only correct result is using 1 processor. The new readin<br>> array is<br>> > identical to the old array.<br>> ><br>> > I am not sure if there is anything wrong on the compiler or any
<br>> > problem. It looks fine when I use gcc to compile. Thank you so<br>> much<br>> > for your help!<br>> ><br>> > Yu-heng<br>> > ---------------------------------------------------<br>> > Yu-Heng Tseng
<br>> ><br>> > Computational Research Division<br>> > Lawrence Berkeley National Laboratory<br>> > One Cyclotron Rd, MS: 50F-1650<br>> > Berkeley, CA94720<br>> > <a href="mailto:YHTseng@lbl.gov">
YHTseng@lbl.gov</a><br>> > 510.495.2904<br>> ><br>> > ----- Original Message -----<br>> > From: Rob Ross <<a href="mailto:rross@mcs.anl.gov">rross@mcs.anl.gov</a>><br>> > Date: Wednesday, April 12, 2006 7:28 pm
<br>> > Subject: Re: Inconsistent results on bluegene<br>> ><br>> >> Hi,<br>> >><br>> >> Can you describe how the results are inconsistent?<br>> >><br>> >> Thanks,<br>
> >><br>> >> Rob<br>> >><br>> >> Yu-Heng Tseng wrote:<br>> >>> Hi,<br>> >>><br>> >>> I am doing a testing on bluegene using. Surprisely, the<br>> results
<br>> >> are not<br>> >>> consistent when I use<br>> >>> multiple processors. There is no error when I compile and<br>> >> install the<br>> >>> library I believe.<br>
> >>> I use the test case from /fandc/pnf_test.F<br>> >>><br>> >>> The compiler I am using is "blrts_xlc" I am not sure what<br>> cause<br>> >> the<br>> >>> problem.
<br>> >>> But when I use gcc to compile it. The results are consistent<br>> for<br>> >>> multiple processors.<br>> >>> Please give me some advices! Thanks!<br>> >>><br>> >>> Yu-heng
<br>> >>><br>> ><br>><br>><br><br></blockquote></div><br>