<div>In the 'change liost' of the new 106 release, I see thie item:</div> <div> </div> <LI>Bugfix for shm and ssm channels. Added missing read and write memory barriers for x86, and missing volatile in packet structure </LI> <div> </div> <div>does it means this problem is fixed ?</div> <div> </div> <div>thanks</div> <div><BR><BR><B><I>William Gropp <gropp@mcs.anl.gov></I></B> wrote:</div> <BLOCKQUOTE class=replbq style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #1010ff 2px solid">We're looking at it; I've added a variation of this to our regular tests. No solution yet, however. My guess is that there is a missing volatile or memory barrier somewhere; this should force us to clean up the current code. <DIV><BR class=khtml-block-placeholder></DIV> <DIV>Bill</DIV> <DIV><BR> <DIV> <DIV>On May 16, 2007, at 12:18 PM, chong tan wrote:</DIV><BR class=Apple-interchange-newline> <BLOCKQUOTE type="cite"><SPAN
class=Apple-style-span style="WORD-SPACING: 0px; FONT: 12px Helvetica; TEXT-TRANSFORM: none; COLOR: rgb(0,0,0); TEXT-INDENT: 0px; WHITE-SPACE: normal; LETTER-SPACING: normal; BORDER-COLLAPSE: separate; border-spacing: 0px 0px; orphans: 2; widows: 2; -khtml-text-decorations-in-effect: none; -apple-text-size-adjust: auto"> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">No taker on this ? There is an identical proble on Linux. Just that I am not sure if this code can reproduce the problem. </SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">tan</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><BR style="FONT-SIZE: 16px;
FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> </SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">----- Original Message ----</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">From: chong tan <<A href="mailto:chong_guan_tan@yahoo.com">chong_guan_tan@yahoo.com</A>></SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">To: <A href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</A></SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">Sent: Friday, April 27, 2007 3:24:09 PM</SPAN><BR style="FONT-SIZE:
16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">Subject: Re: [MPICH] MPICH105 shm drops packages on SUN niagara</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">The following code reproduces the problem. I think you maybe able to reproduce the error on</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">Linux, but I am not sure.</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"></SPAN> </DIV> <DIV
style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"></SPAN> </DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">It is best to run :</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">mpiexec -n 8 a.out</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">to reproduce the problem. You will need a machine with 8CPU/cores. SOmetime you will need to</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">run the code multiple time to see the error.</SPAN></DIV> <DIV
style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"></SPAN> </DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">there will be files fast_mpi_?.dmp created, where ? is the rank of the related 'rank'. When MPI get stuck,</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">you should look at the last line of fast_mpi_0.dmp. If it says:</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> read from child 7</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"></DIV> <DIV style="FONT-SIZE: 16px;
FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"></SPAN> </DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">then you should look at the last line of fast_mpi_7.dmp, it will say:</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> read from master</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"></SPAN> </DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">hope this help to debug the error.</SPAN></DIV> <DIV style="FONT-SIZE: 16px;
FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"></SPAN> </DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">thanks</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">tan</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">---------------------</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">#include "stdlib.h"</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY:
times new roman">#include "stdio.h"</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">#include "mpi.h"</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"></SPAN> </DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">#define LOOP_COUNT 1000000</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">#define DATA_SIZE 4</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">#define MP_TAG 999</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY:
times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">main()</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">{</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> int nProc, rank ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> int argc = 0 ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> int i, j, status ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span
style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> char buf[ 128 ] ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> FILE *pf ;</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> MPI_Init( &argc, NULL ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> MPI_Comm_size( MPI_COMM_WORLD, &nProc ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> MPI_Comm_rank( MPI_COMM_WORLD, &rank ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times
new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> sprintf( buf, "fast_mpi_%d.dmp", rank ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> pf = fopen( buf, "w" ) ;</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> if( !rank ) {</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> int **psend ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> int
**precv ;</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> psend = (int**)calloc( nProc, sizeof( int *) ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> precv = (int**)calloc( nProc, sizeof( int *) ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> for( i = 0 ; i < nProc ; i++ ) {</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> psend[ i ] = (int*)calloc( DATA_SIZE, sizeof( int ) ) ;</SPAN><BR
style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> precv[ i ] = (int*)calloc( DATA_SIZE, sizeof( int ) ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> }</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> for( i = 0 ; i < LOOP_COUNT ; i++ ) {</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fprintf( pf, "Master : loop %d\n", i ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN
class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fflush( pf ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> for( j = 1 ; j < nProc ; j++ ) {</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fprintf( pf, " read from child %d\n", j ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fflush( pf ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span
style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> status = MPI_Recv( precv[ j ], DATA_SIZE, MPI_LONG, j, MP_TAG, MPI_COMM_WORLD, MP</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">I_STATUS_IGNORE ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fprintf( pf, " read from child %d done, status = %d\n", j, status ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fflush( pf ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN
class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> }</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> for( j = 1 ; j < nProc ; j++ ) {</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fprintf( pf, " send to child %d\n", j ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fflush( pf ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE:
16px; FONT-FAMILY: times new roman"> status = MPI_Send( psend[ j ], DATA_SIZE - 1, MPI_LONG, j, MP_TAG, MPI_COMM_WORLD</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fprintf( pf, " send to child %d done, status = %d\n", j, status ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fflush( pf ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE:
16px; FONT-FAMILY: times new roman"> }</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> }</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> } else {</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> int *psend ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> int *precv ;</SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px;
FONT-FAMILY: times new roman"> psend = (int*)calloc( DATA_SIZE, sizeof( int ) ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> precv = (int*)calloc( DATA_SIZE, sizeof( int ) ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> for( i = 0 ; i < LOOP_COUNT ; i++ ) {</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fprintf( pf, " send to master\n" ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new
roman"> fflush( pf ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> status = MPI_Send( psend, DATA_SIZE - 1, MPI_LONG, 0, MP_TAG, MPI_COMM_WORLD ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fprintf( pf, " send to master done, status = %d\n", status ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fflush( pf ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new
roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fprintf( pf, " read from master\n" ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fflush( pf ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> status = MPI_Recv( precv, DATA_SIZE, MPI_LONG, 0, MP_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fprintf(
pf, " read from master done, status = %d\n", status ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fflush( pf ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> }</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> }</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> fclose( pf ) ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new
roman"> MPI_Finalize() ;</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">}</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"></DIV></DIV><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"> <HR SIZE=1> <SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">Ahhh...imagining that irresistible "new car" smell?</SPAN><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman">Check out </SPAN><A href="http://us.rd.yahoo.com/evt=48245/*http://autos.yahoo.com/new_cars.html;_ylc=X3oDMTE1YW1jcXJ2BF9TAzk3MTA3MDc2BHNlYwNtYWlsdGFncwRzbGsDbmV3LWNhcnM-" target=_blank rel=nofollow><SPAN class=Apple-style-span style="FONT-SIZE: 16px; COLOR: rgb(0,0,238); FONT-FAMILY: times new roman; -khtml-text-decorations-in-effect: underline">new cars at
Yahoo! Autos.</SPAN></A><SPAN class=Apple-style-span style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"></SPAN></DIV> <DIV style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"><BR style="FONT-SIZE: 16px; FONT-FAMILY: times new roman"></DIV></DIV><BR> <HR SIZE=1> Be a better Heartthrob.<SPAN class=Apple-converted-space> </SPAN><A href="http://us.rd.yahoo.com/evt=48255/*http://answers.yahoo.com/dir/_ylc=X3oDMTI5MGx2aThyBF9TAzIxMTU1MDAzNTIEX3MDMzk2NTQ1MTAzBHNlYwNCQUJwaWxsYXJfTklfMzYwBHNsawNQcm9kdWN0X3F1ZXN0aW9uX3BhZ2U-?link=list&sid=396545433"><SPAN class=Apple-style-span style="COLOR: rgb(0,0,238); -khtml-text-decorations-in-effect: underline">Get better relationship answers </SPAN></A>from someone who knows.<BR>Yahoo! Answers - Check it out.</SPAN></BLOCKQUOTE></DIV><BR></DIV></BLOCKQUOTE><BR><p> 
<hr size=1>Luggage? GPS? Comic books? <br>
Check out fitting <a href="http://us.rd.yahoo.com/evt=48249/*http://search.yahoo.com/search?fr=oni_on_mail&p=graduation+gifts&cs=bz"> gifts for grads</a> at Yahoo! Search.