Hints on improving performance with WRF and Pnetcdf

[Craig Tierney]

On 9/4/10 8:25 PM, Gerry Creager wrote:
> Rob Latham wrote:
>> On Thu, Sep 02, 2010 at 06:23:42PM -0600, Craig Tierney wrote:
>>> I did try setting the hints myself by changing the code, and performance
>>> still stinks (or is no faster). I was just looking for a way to not
>>> have to modify WRF, or more importantly have every user modify WRF.
>>
>> What's going slowly?
>> If wrf is slowly writing record variables, you might want to try
>> disabling collective I/O or carefully selecting the intermediate
>> buffer to be as big as one record.
>>
>> That's the first place I'd look for bad performance.
>
> Ah, but I'm seeing the same thing on Ranger (UTexas). I'm likely going
> to have to modify the WRF pnetcdf code to identify a sufficiently large
> stripe count (Lustre file system) to see any sort of real improvement.
>
> More to the point, I see worse performance than with normal Lustre and
> regular netcdf. AND, there's no way to set MPI-IO-HINTS in the SGE as
> configured on Ranger. We've tried and their systems folk concur, so it's
> not just me saying it.
>

What do you mean you can't?  How would you set it in another batch system?

> I will look at setting the hints file up but I don't think that's going
> to give me the equivalent of 64 stripe counts, which looks like the
> sweet spot for the domain I'm testing on.
>

So what Hints are you passing and is then the key to increase the number
of stripes for the directory?

> Craig, one I have time to get back on to this, I think we can convince
> NCAR to add this as a bug release. I also anticipate the tweak will be
> on the order of 4-5 lines.
>

I already wrote code so that if you set the variable WRF_MPIIO_HINTS, 
and list all the hints you want to set (comma delimited), then the code 
in external/io_pnetcdf/wrf_IO.F90 will set the hints for you.  When
I see that any of this actually helps I will send the patch in for 
future use.

Craig