[Swift-user] Problems in complex mapping situation

[Michael Wilde]

Cool. I forgot about this. I can certainly try it for perf measurements.

A mapper-like external script can walk the output tree of the concurrent 
mapper and create links with the desired filenames.  I'll try that.

Some questions (but I'll find this out in a moment when I try it):

- does this behavior start as soon as the directory hits 50 elements?
- if the first element I store is, say 1000, is it triggered?
- does it map arrays of structures?

Im assuming yes to all these, and will experiment. Thanks!

On 3/20/09 4:15 AM, Ben Clifford wrote:
> On Fri, 20 Mar 2009, Michael Wilde wrote:
> 
>> This script was working fine, and I was trying to improve its performance on
>> large datasets on the BG/P by using an ext mapper instead of simple_mapper to
>> map a large 2D array of structures (OOPSOut result) so that it spreads across
>> multiple directories (to avoid the GPFS locking issue. (I have 4K cores
>> writing 14,000 files to one directory, as I see no way with simple mapper to
>> use the array index to, for example, insert more directory entries in the
>> prefix or suffix.)
> 
> The concurrent mapper (used to map variables that have no explicitly 
> declared mapper) makes a tree of directories for large arrays.
> 
> This is done in a deliberately unspecified manner, but at present 
> (according to the source)  /** determines how many directories and element 
> files are permitted
>             in each directory. There will be no more than
>             DIRECTORY_LOAD_FACTOR element files and no more than
>             DIRECTORY_LOAD_FACTOR directories, so there could be up to
>             2 * DIRECTORY_LOAD_FACTOR elements. */
>         public final static int DIRECTORY_LOAD_FACTOR=25;
> 
> so if you declare an array as  myarray foo[]; with no mapper, you'll end 
> up with a tree of directories, each directory having no more than 50 
> entries in it.
> 
> You lose the ability to specify the filenames at all here, which may or 
> may not be a problem for your applications.
> 
> It might be that this functionality could be more made more general so 
> that you can (for example) specify some option to the simple mapper to get 
> automatically-computed hierarchical directories. It seems like a common 
> enough use case.
>