<br><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Akram Smaali</b> <span dir="ltr"><<a href="mailto:akram.smaali@gmail.com">akram.smaali@gmail.com</a>></span><br>
Date: 2012/7/12<br>Subject: Re: [mpich-discuss] Memory leaks using ROMIO<br>To: Jayesh Krishna <<a href="mailto:jayesh@mcs.anl.gov">jayesh@mcs.anl.gov</a>><br><br><br><div>I think that, as Rob latham said, it's caused by ntfs. I was looking into the source code of ROMIO but i didn't see anything indicating that. I suppose that the OS doesn't get back the allocated buffer from the user space. </div>
<div class="im">
<div><br></div># Does the system memory decrease after a period of time? </div><div>No, it does just at the end. </div><div class="im"><div># Does the memory increase correspond to the amount of data written?
</div></div><div>Yes excactly the amount of data written.<div class="im"><br># How much does the memory increase? </div></div><div>i use a 5GB file so it increase that much.</div><div class="im"><div># Do you see similar behavior with a simple Windows program (CreateFileMapping() + WriteFile() + CloseHandle())? </div>
</div><div>It just happens with the MPI_File_read_at()</div><div><div><div class="h5"><br><div class="gmail_quote">2012/7/12 Jayesh Krishna <span dir="ltr"><<a href="mailto:jayesh@mcs.anl.gov" target="_blank">jayesh@mcs.anl.gov</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
Windows is known to delay writes even after the user is done with the file (I have seen similar issues before. Your problem could be different though.).<br>
<br>
# Does the system memory decrease after a period of time?<br># Does the memory increase correspond to the amount of data written? <br>
# How much does the memory increase?<br>
# Do you see similar behavior with a simple Windows program (CreateFileMapping() + WriteFile() + CloseHandle())?<br>
<br>
Regards,<br>
Jayesh<br>
<div><div><br>
----- Original Message -----<br>
From: "Rob Latham" <<a href="mailto:robl@mcs.anl.gov" target="_blank">robl@mcs.anl.gov</a>><br>
To: <a href="mailto:mpich-discuss@mcs.anl.gov" target="_blank">mpich-discuss@mcs.anl.gov</a><br>
Sent: Thursday, July 12, 2012 11:09:50 AM<br>
Subject: Re: [mpich-discuss] Memory leaks using ROMIO<br>
<br>
On Thu, Jul 12, 2012 at 01:29:38PM +0200, Akram Smaali wrote:<br>
> Hello,<br>
><br>
> i'm using mpich2 in a C program that reads several buffers from a data text<br>
> files.<br>
> I noticed that even when i free the buffer at the end, the global memory<br>
> usage still increase but the memory usage per process still the same.<br>
> I changed the MPI_File_read_at() with fseek/fread and the problem has<br>
> dissapeared.<br>
> I'm wondering if that i should do some extra instriuctions to avoid that<br>
> because it seems that mpi_read allocate an intermediate buffer for reading<br>
> (sieving optimization maybe) from the user space memory and the OS (<br>
> windosws 7) don't get it back.<br>
> i'm using MPICH2 1.4 on w windows 7 with NTFS file system.<br>
<br>
It might be an NTFS-specific leak. I don't see any leaks on my linux<br>
system (ad_ufs).<br>
<br>
> Thank you for your help.<br>
><br>
> here is my code :<br>
<br>
It would be great if you could make this case more self contained.<br>
Whatever offsets are in your Dictionary file, put those in an array<br>
and broadcast that.<br>
<br>
> #include <stdio.h><br>
> #include <stdlib.h><br>
> #include <mpi.h><br>
> #include <omp.h><br>
> #include <string.h><br>
> #include <time.h><br>
> #include "queue.h"<br>
> #include "KhiopsNativeInterface.h"<br>
> #include <windows.h><br>
><br>
> #define maxLength 100000<br>
><br>
> double CPUtime(){ return ((double) clock())/CLOCKS_PER_SEC;}<br>
><br>
><br>
> int main(int argc, char* argv []){<br>
><br>
> if(argc != 5) {<br>
> printf("\t[Dictionary file] [Dictionary] [Input file] [Buffer size]\n");<br>
> exit(0);<br>
> }<br>
><br>
><br>
><br>
> char* sDictionaryFileName = argv[1];<br>
> char* sDictionaryName = argv[2];<br>
> char* filename = argv[3];<br>
> int Mbuffer = atoi(argv[4]);<br>
><br>
> int maxBuffer = Mbuffer*1024*1024;<br>
> int over = 10000;<br>
><br>
> int rank,numprocess;<br>
> long int offset;<br>
><br>
> char* buffer;<br>
> char* opbuffer;<br>
><br>
> double tstart=CPUtime();<br>
><br>
> MPI_Init( &argc, &argv );<br>
> MPI_Comm_rank( MPI_COMM_WORLD, &rank );<br>
><br>
> /* mpi version */<br>
> /* open the file*/<br>
> MPI_File fh;<br>
> int err;<br>
> err = MPI_File_open(MPI_COMM_WORLD, filename, MPI_MODE_RDONLY,<br>
> MPI_INFO_NULL, &fh);<br>
> if (err != MPI_SUCCESS) {<br>
> char errstr[MPI_MAX_ERROR_STRING];<br>
> int errlen;<br>
> MPI_Error_string(err, errstr, &errlen);<br>
> printf("Error at opening file %s (%s)\n",filename,errstr);<br>
> MPI_Finalize();<br>
> exit(1);<br>
> }<br>
> // get offsets and buffer size<br>
> MPI_Offset sfile;<br>
> MPI_File_get_size(fh,&sfile);<br>
> MPI_Status status;<br>
><br>
><br>
> /* C version */<br>
> /*FILE* fh;<br>
> long int sfile;<br>
> fh =fopen( filename,"rb");<br>
> if (fh==NULL) {<br>
> printf("Error at opening file %s\n",filename);<br>
> exit(1);<br>
> }<br>
> // get offsets and buffer size<br>
> fseek(fh, 0L, SEEK_END);<br>
> sfile = ftell(fh);<br>
> fseek(fh, 0L, SEEK_SET);*/<br>
><br>
><br>
> MPI_Comm_size( MPI_COMM_WORLD, &numprocess );<br>
><br>
> /* number of iterations */<br>
> long int data_size = (long int)(sfile/(numprocess));<br>
> int nbIter = data_size/maxBuffer;<br>
> if(nbIter<=1){<br>
> nbIter = 1;<br>
> maxBuffer = data_size;<br>
> }<br>
><br>
> /* offsets */<br>
> offset = data_size*(rank);<br>
> long int cursor = offset;<br>
> char* header;<br>
> if(rank==0){<br>
> FILE* fh;<br>
> fh =fopen( filename,"rb");<br>
> if (fh==NULL) {<br>
> printf("Error at opening file %s\n",filename);<br>
> exit(1);<br>
> }<br>
> /* read the header and broadcast it */<br>
> header = malloc(sizeof(char)*1000);<br>
> fgets(header,1000,fh);<br>
> fclose(fh);<br>
><br>
> //broadcast header<br>
> int sndHeader = strlen(header);<br>
> //cursor+=sndHeader;<br>
> int process_counter;<br>
> for(process_counter=1;process_counter<numprocess;process_counter++){<br>
> int ierr = MPI_Send(&sndHeader,sizeof(int), MPI_INT, process_counter,<br>
> 42,MPI_COMM_WORLD);<br>
> if (ierr != MPI_SUCCESS) {<br>
> int errclass,resultlen;<br>
> char err_buffer[MPI_MAX_ERROR_STRING];<br>
> MPI_Error_class(ierr,&errclass);<br>
> if (errclass== MPI_ERR_RANK) {<br>
> fprintf(stderr,"Invalid rank used in MPI send call\n");<br>
> MPI_Error_string(ierr,err_buffer,&resultlen);<br>
> fprintf(stderr,err_buffer);<br>
> MPI_Finalize();<br>
> }<br>
> }<br>
> MPI_Send(header, sndHeader, MPI_CHAR, process_counter, 43, MPI_COMM_WORLD);<br>
> }<br>
> }<br>
> else{<br>
> /* receive the header */<br>
> int sizeofHeader;<br>
> MPI_Status s ;<br>
> MPI_Recv(&sizeofHeader,sizeof(int),MPI_INT,0,42,MPI_COMM_WORLD,&s);<br>
> header = malloc (sizeof(char)*sizeofHeader+1);<br>
> MPI_Recv(header,sizeofHeader,MPI_CHAR,0,43,MPI_COMM_WORLD,&s);<br>
> }<br>
><br>
><br>
><br>
> /* Synchronization barrier */<br>
> MPI_Barrier(MPI_COMM_WORLD);<br>
><br>
> int count;<br>
><br>
> opbuffer = malloc(sizeof(char)*maxBuffer);<br>
><br>
> /* C version */<br>
> //fseek(fh,cursor,SEEK_SET);<br>
><br>
> for(count=0;count<nbIter;count++){<br>
><br>
> if(count==0 && rank==numprocess-1){ //init ring<br>
> //send the token to p0<br>
> int token=1;<br>
> MPI_Send(&token,sizeof(int),MPI_INT,0,55,MPI_COMM_WORLD);<br>
> }<br>
><br>
> //recv<br>
> int token;<br>
> int sender;<br>
> if(rank==0)<br>
> sender = numprocess-1;<br>
> else<br>
> sender=rank-1;<br>
><br>
> MPI_Status s;<br>
> MPI_Recv(&token,sizeof(int),MPI_INT,sender,55,MPI_COMM_WORLD,&s);<br>
> fflush(stdout);printf("P%d got the token at %G\n",rank,CPUtime());<br>
> //read<br>
> double start=CPUtime();<br>
> /*double readtime;<br>
> double sread=CPUtime();//read time*/<br>
><br>
> //read<br>
> if(token==1){<br>
> /* MPI version */<br>
> int err=MPI_File_read_at(fh, cursor,opbuffer, sizeof(char)*maxBuffer,<br>
> MPI_CHAR, &status);<br>
> if(err!=MPI_SUCCESS){<br>
> char errstr[MPI_MAX_ERROR_STRING];<br>
> int errlen;<br>
> MPI_Error_string(err, errstr, &errlen);<br>
> printf("Error reading file %s (%s)\n",filename,errstr);<br>
> MPI_Finalize();<br>
> exit(0);<br>
> }<br>
><br>
> /* C version of read */<br>
> /*int k=fread(opbuffer,sizeof(char),maxBuffer,fh);<br>
> if(k==0)<br>
> perror("fread");*/<br>
><br>
> cursor+=maxBuffer;<br>
> buffer=opbuffer;<br>
><br>
> }<br>
> else{<br>
> printf("Error token!\n");<br>
> token=1;<br>
> }<br>
> //printf("P%d readtime=%G\n",rank,CPUtime()-sread);<br>
> //Isend<br>
> int next = (rank+1)%numprocess;<br>
> MPI_Send(&token,sizeof(int),MPI_INT,next,55,MPI_COMM_WORLD);<br>
><br>
> //i perform processing buffer here!<br>
><br>
> }<br>
> free(opbuffer);<br>
><br>
> /* mpi version */<br>
> int er=MPI_File_close(&fh);<br>
> if(er!=MPI_SUCCESS){<br>
> printf("Error closing file\n");<br>
> MPI_Finalize();<br>
> exit(1);<br>
> }<br>
><br>
> /* c version */<br>
> //fclose(fh);<br>
><br>
><br>
> MPI_Finalize();<br>
><br>
> printf("Global time : %G\n",CPUtime()-tstart);<br>
> return 0;<br>
> }<br>
><br>
><br>
> Best Regards<br>
<br>
> _______________________________________________<br>
> mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov" target="_blank">mpich-discuss@mcs.anl.gov</a><br>
> To manage subscription options or unsubscribe:<br>
> <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
<br>
<br>
--<br>
Rob Latham<br>
Mathematics and Computer Science Division<br>
Argonne National Lab, IL USA<br>
_______________________________________________<br>
mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov" target="_blank">mpich-discuss@mcs.anl.gov</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div dir="ltr"><div><div dir="ltr"><b>Akram SMAALI</b><br><div><font size="1" color="#000066">Ingénieur stagiaire à France Télécom R&D</font></div>
<div><font size="1" color="#000066">22300 Lannion, France</font></div><div><font size="1" color="#000066"><b>mobile: <a href="tel:%2B33%207%2051%2013%2039%2011" value="+33751133911" target="_blank">+33 7 51 13 39 11</a> </b></font></div>
<div><font size="1"><font color="#000066"><b>mail:</b></font> <a href="mailto:akram.smaali@gmail.com" target="_blank">akram.smaali@gmail.com</a> </font></div>
<div><font size="1"><br></font></div></div></div></div><br>
</font></span></div>
</div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div><div dir="ltr"><b>Akram SMAALI</b><br><div><font size="1" color="#000066">Ingénieur stagiaire à France Télécom R&D</font></div><div><font size="1" color="#000066">22300 Lannion, France</font></div>
<div><font size="1" color="#000066"><b>mobile: +33 7 51 13 39 11 </b></font></div><div><font size="1"><font color="#000066"><b>mail:</b></font> <a href="mailto:akram.smaali@gmail.com" target="_blank">akram.smaali@gmail.com</a> </font></div>
<div><font size="1"><br></font></div></div></div></div><br>