[Swift-devel] [Bug 143] New: scalability limitation in concurrent mapper used for local variables with GPFS-like site-shared filesystem

[bugzilla-daemon at mcs.anl.gov]

http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=143

           Summary: scalability limitation in concurrent mapper used for
                    local variables with GPFS-like site-shared filesystem
           Product: Swift
           Version: unspecified
          Platform: Macintosh
        OS/Version: Mac OS
            Status: NEW
          Severity: normal
          Priority: P2
         Component: General
        AssignedTo: benc at hawaga.org.uk
        ReportedBy: benc at hawaga.org.uk
                CC: swift-devel at ci.uchicago.edu


The concurrent mapper has a GPFS-friendly scalability feature for keeping large
numbers of files out of the same directory, based on splitting up array indices
into a multiple tier directory tree.

However, it does not do this for the thread identifier.

I think this means that an array of intermediate files declared outside of a
large foreach will exhibit this scalability property, whilst an intermediate
variable declared inside the loop will have all of its files placed in one
directory.

This is probably a scalability limitation on GPFS when local variables are used
in loops which will manifest as poor performance in such circumstances.


-- 
Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.