pnetcdf with CDO

[Latham, Robert J.]

On Mon, 2014-02-24 at 21:16 +0000, Liu, Jaln wrote:
> Hi Guys,
> 
> We are planning to implement a CDO-like tool using PnetCDF,
> The CDO is short for climate data operator, which provides more than 200 operators to manipulate the climate dataset, but few parallel support,(about 20 operators with openmp support).
> Can anyone of you please provide any suggestion?

Yeah!  Check out the ParViz/ParGAL/ParNCL project:

http://trac.mcs.anl.gov/projects/parvis/

ParNCL has the nice feature of being a scripting language: the
description of what's going to happen and the execution happen in the
same environment.

You can sort of do that with CDO, but the most common workflow we found
for CDO was "extract this variable to a temporary file followed by
"compute weighted average of the variable in this temporary file".

You can do parallelism on a per-command basis, but you will be limited
to maybe order-100 way parallelism if you do so.  The real parallelism
is over the entire workflow, where this "compute weighted average" has
to happen for 300-some history steps.

It's been a few years since we wrote the proposal for ParVis: we dropped
"ParNCO" because it was a lot more clear how to introduce parallelism
into NCL. 

==rob