Burst buffer - blocking or unblocking transfer?

[Latham, Robert J.]

On Tue, 2020-09-22 at 18:29 +0000, Michael Laufer wrote:
> Hi,
>  
> In reference to the burst buffer feature introduced in v1.10.0:
> When the burst buffer flushes to the (long term storage) disk, does
> it do so in a blocking or unblocking fashion?
>  
> It appears to me that it is blocking, but I am not 100% sure. If that
> is the case, why not use an unblocking (async) transfer?
> This would allow the computation to continue while the data transfer
> from BB to disk in running.
>  
> Please let me know if I am missing something.
> Michael Laufer

Parallel-NetCDF could definitely issue MPI_File_iwrite calls, but with
what would it overlap that I/O?  The replay from burst buffer log to
stable storage happens when pnetcdf closes a file, waits for operations
to complete, flushes data, or finds a read.

In the first three cases, there is no operation we can overlap with the
write.

In the last case, we wait for the logs to replay so we read back data
as the application expects it.

The big benefit for using the burst buffer feature is to soak up tiny
noncontiguous writes.

==rob