<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000'><br><br><hr id="zwchr"><blockquote id="DWT257" style="border-left:2px solid rgb(16, 16, 255);margin-left:5px;padding-left:5px;">On Fri, Oct 19, 2012 at 03:16:46PM -0500, Iulian Grindeanu wrote:<br>> Hello Rob, <br>> I think that change has to happen in src/parallel/ReadParallel.cpp <br>> I am not sure yet though, Tim would confirm that <br><br>Interesting. What is this POPT_BCAST option? <br><br>I don't want to change all of moab into 'rank 0 does i/o' --<br>obviously that's not going to scale. <br><br></blockquote>this is read/broadcast option, which as you said, does not scale.<br><blockquote id="DWT258" style="border-left:2px solid rgb(16, 16, 255);margin-left:5px;padding-left:5px;">But for some of these inputs we are looking at datasets that are not<br>all that big:<br><br>/intrepid-fs0/users/tautges/persistent/meshes/2bricks/nogeom/64bricks_8mtet_ng_rib_2048.h5m<br>has a 7995686 x 4 "connectivity" dataset, but I know from talking with<br>jason that you are only pulling one column out of this array, so 61<br>MiBytes. <br></blockquote>These are the connectivity arrays for ~ 8 million tetra elements. We do not read them on all procs, every processor needs to find out first<br>what subset of elements has to read; the set information should be then enough to decide what portion of the connectivity array needs to be <br>read on each processor. <br><br>Iulian<br><blockquote style="border-left:2px solid rgb(16, 16, 255);margin-left:5px;padding-left:5px;"><br>==rob<br><br>> <br>> ----- Original Message -----<br>> <br>> | Tim knows all this but for the rest of the list, here's the short<br>> | story:<br>> <br>> <br>> | MOAB's HDF5 reader and writer have a problem on BlueGene where it will<br>> <br>> | collectively read in initial conditions or write output, and run out<br>> <br>> | of memory. This out-of-memory condition comes from MOAB doing all the<br>> <br>> | right things -- using HDF5, using collective I/O -- but the MPI-IO<br>> <br>> | library on Intrepid goes and consumes too much memory.<br>> <br>> <br>> | I've got one approach to deal with the MPI-IO memory issue for writes.<br>> <br>> | This approach would sort of work for the reads, but what is really<br>> <br>> | needed is for rank 0 to do the read and broadcast the result to<br>> <br>> | everyone.<br>> <br>> <br>> | So, I'm looking for a little help understanding MOAB's read side of<br>> <br>> | the code. Conceptually, all processes read the table of entities.<br>> <br>> <br>> | A fairly small 'mbconvert' job will run out of memory:<br>> <br>> <br>> | 512 nodes, 2048 processors:<br>> <br>> <br>> | ======<br>> <br>> | NODES=512<br>> <br>> | CORES=$(($NODES * 4))<br>> <br>> | cd /intrepid-fs0/users/robl/scratch/moab-test<br>> <br>> <br>> | cqsub -t 15 -m vn -p SSSPP -e MPIRUN_LABEL=1:BG_COREDUMPONEXIT=1 \<br>> <br>> | -n $NODES -c $CORES /home/robl/src/moab-svn/build/tools/mbconvert\<br>> <br>> | -O CPUTIME -O PARALLEL_GHOSTS=3.0.1 -O PARALLEL=READ_PART \<br>> <br>> | -O PARALLEL_RESOLVE_SHARED_ENTS -O PARTITION -t \<br>> <br>> | -o CPUTIME -o PARALLEL=WRITE_PART<br>> | /intrepid-fs0/users/tautges/persistent/meshes/2bricks/nogeom/64bricks_8mtet_ng_rib_${CORES}.h5m<br>> | \<br>> <br>> | /intrepid-fs0/users/robl/scratch/moab/8mtet_ng-${CORES}-out.h5m<br>> <br>> | ======<br>> <br>> <br>> | I'm kind of stumbling around ReadHDF5::load_file and<br>> <br>> | ReadHDF5::load_file_partial trying to find a spot where a collection<br>> <br>> | of tags are read into memory. I'd like to, instead of having all<br>> <br>> | processors do the read, have just one processor read and then send the<br>> <br>> | tag data to the other processors.<br>> <br>> <br>> | First, do I remember the basic MOAB concept correctly: that early on<br>> <br>> | every process reads the exact same tables out of the (in this case<br>> <br>> | HDF5) file?<br>> <br>> <br>> | If I want rank 0 to do all the work and send data to other ranks,<br>> <br>> | where's the best place to slip that in? It's been a while since I did<br>> <br>> | anything non-trivial in C++, so some of these data structures are kind<br>> <br>> | of greek to me.<br>> <br>> <br>> | thanks<br>> <br>> | ==rob<br>> <br>> <br>> | --<br>> <br>> | Rob Latham<br>> <br>> | Mathematics and Computer Science Division<br>> <br>> | Argonne National Lab, IL USA<br>> <br>> <br>> <br><br>-- <br>Rob Latham<br>Mathematics and Computer Science Division<br>Argonne National Lab, IL USA<br></blockquote><br></div></body></html>