performance issue

[Wei-Keng Liao]

I found results of using GPFS from one of your previous emails.

> GPFS (/glade/work on derecho):
> RESULT: write    SUBSET         1        16        64     4570.2078677815        4.0610844270
> RESULT: write    SUBSET         1        16        64     4470.3231494386        4.1518251320


I checked the ROMIO source codes. The codes for copying data look the same
to me, although they are not shared among different file system drivers.
Strange.

Wei-keng 

> On Aug 14, 2023, at 2:11 PM, Wei-Keng Liao <wkliao at northwestern.edu> wrote:
> 
> Did you run the same tests on a non-Lustre file system and see no difference?
> Can you show me the timings? 
> 
> Wei-keng 
> 
>> On Aug 14, 2023, at 11:54 AM, Jim Edwards <jedwards at ucar.edu> wrote:
>> 
>> Hi Wei-Keng,
>> 
>> Thanks for looking into this.   Because the allocations in pioperformance.F90 are done on the compute nodes and
>> not the IO nodes I don't think that your suggestion would make any difference.   I also wonder why this issue 
>> appears to be so specific to the lustre file system - presumably the ROMIO functionality you speak of is general and not
>> specific to lustre?   Anyway your analysis spurred me to try something else which seems to work: prior to calling
>> ncmpi_iput_varn in pio_darray_int.c I added a call to ncmpi_wait_all to make sure that any existing buffer was written.
>> 
>> This seems to have fixed the problem and my writes are now
>> RESULT: write    SUBSET         1        16        64     4787.6393342631        3.8766495770
>> RESULT: write    SUBSET         1        16        64     4803.9296372205        3.8635037150
>> 
>