[mpich-discuss] MPI-IO and (potentially) unnecessary synchronization

[burlen]

Could anyone explain  why MPI_File_close must be a collective call when 
collective buffering is not used? By collective I mean block the 
progress of each process until at least all of the processes have 
entered the call?

I realize my first post misunderstands the situation in a number of 
ways. To attempt to correct myself, each process who touches the disk 
must have his own file descriptor somewhere. When collective buffering 
isn't used to close the file each process would have to close his local 
descriptor. I have noticed that MPI_File_sync is documented as a 
collective function but does not behave like one when collective 
buffering is not used. By this I mean that it completes before all 
processes have entered the call. if MPI_File_sync can behave this way, 
why wouldn't MPI_File_close do the same?

burlen wrote:
> in benchmarks of very large concurrent writes on Lustre using both cb 
> and non cb API I have observed that for the non cb API that 
> asynchronism during write can be advantageous as it tends to reduce 
> congestion and contention. This can increase the throughput.  However, 
> in this case the synchronization time that occurs at MPI_File_close is 
> significant for many of the processes, as non return until the slowest 
> process enters the call. This synchronization at close in net effect 
> ruins any advantage gained. So I wonder does MPI_File_close really 
> require a collective implementation? For instance I could imagine a 
> reference counting scheme where one process were designated to manage 
> close operation, others call MPI_File_sync (which I've observed 
> doesn't block unless it has to) and post a non-blocking 0 byte message 
> to the manager rank, then they can continue unimpeded. You could 
> perhaps remove all to one communication using some sort of 
> hierarchical structured communication pattern. If I understand such a 
> scheme wouldn't violate consistency because if one cares about it then 
> a barrier is required anyway.
>
> Have I misunderstood the situation?
>
> Thanks
> Burlen
>