[MOAB-dev] mixed mesh question

Iulian Grindeanu iulian at mcs.anl.gov
Tue Nov 19 15:33:58 CST 2013


OK, that was it; 
I made SetStruct a class, initialized those members data properly (in the constructor), and it works now. 
That test I was running is fine now. 

Thanks again, 
Iulian 

----- Original Message -----

| Hi Carlos and Junior,
| Thank you again for the cgns reader/writer.
| I think I am close to integrate it, at least in 462.
| After I changed the writer hpp file (), it built, but I see some
| issues when I am testing.
| Could be related to my changes, probably I should initialize those
| members in a constructor.

| So this test (reader) that I am doing seems fine:
| ../../../build46/tools/mbsize 2d_naca0012.cgns
| File 2d_naca0012.cgns:

| number of nodes = 3555
| number of elems = 6944
| number of parts = 3
| type count total minimum average rms maximum std.dev.
| ------- ----- ------- -------------------- --------------------
| -------------------- -------------------- --------------------
| Edge 166 65 0.0061783 0.39047 0.71774 2.0409 0.60223
| Tri 6944 3.1e+02 2.2192e-05 0.045078 0.13326 1.4755 0.1254
| 1D Side 20832 3.9e+03 0.0061783 0.18654 0.33264 2.0409 0.27542
| Vertex 3555

| But this (reader and writer) has issues. Again, it could be my
| changes, to make it compile.

| ../../../build46/tools/mbconvert 3d_wingbody_mixed_JR.cgns test.cgns
| *** Warning:The file being read is more recent that the CGNS library
| used ***

| number of nodes = 285905
| number of elems = 333820
| number of parts = 1
| Read "3d_wingbody_mixed_JR.cgns"
| THE CGNS CONVERSION ONLY WORKS FOR ENTITIES CITED BELOW:
| -MBVERTEX
| -MBEDGE
| -MBTRI
| -MBQUAD
| -MBTET
| -MBPYRAMID
| -MBHEX

| The Number of Vertex is 285905.
| The Number of Edges is 0.
| The Number of Faces is 29782.
| The Number of Cells is 333820.

| test.cgns file is a 3-D mesh.

| The mesh has 5 Tags.
| Tag name= MATERIAL_SET
| Number of MBEDGE = 0
| Number of MBTRI = 0
| Number of MBQUAD = 0
| Number of MBTET = 0
| Number of MBPYRAMID = 0
| Number of MBPRISM = 0
| Number of MBHEX = 0
| Total number of Edges = 0
| Total number of Faces = 0
| Total number of Cells = 0
| Sets[0].CGNSType= 0
| Sets[0].IdSet = 0

| Tag name= NEUMANN_SET
| Number of MBEDGE = 0
| Number of MBTRI = 0
| Number of MBQUAD = 0
| Number of MBTET = 0
| Number of MBPYRAMID = 0
| Number of MBPRISM = 0
| Number of MBHEX = 0
| Total number of Edges = 0
| Total number of Faces = 0
| Total number of Cells = 0
| Sets[1].CGNSType= 0
| Sets[1].IdSet = 0

| Tag name= DIRICHLET_SET
| Number of MBEDGE = 0
| Number of MBTRI = 0
| Number of MBQUAD = 0
| Number of MBTET = 0
| Number of MBPYRAMID = 0
| Number of MBPRISM = 0
| Number of MBHEX = 0
| Total number of Edges = 0
| Total number of Faces = 0
| Total number of Cells = 0
| Sets[2].CGNSType= 0
| Sets[2].IdSet = 0

| Tag name= GEOM_DIMENSION
| Number of MBEDGE = 0
| Number of MBTRI = 0
| Number of MBQUAD = 0
| Number of MBTET = 0
| Number of MBPYRAMID = 0
| Number of MBPRISM = 0
| Number of MBHEX = 0
| Total number of Edges = 0
| Total number of Faces = 0
| Total number of Cells = 0
| Sets[3].CGNSType= 0
| Sets[3].IdSet = 0

| Tag name= GLOBAL_ID
| Number of MBEDGE = 0
| Number of MBTRI = 0
| Number of MBQUAD = 29782
| Number of MBTET = 72460
| Number of MBPYRAMID = 3836
| Number of MBPRISM = 0
| Number of MBHEX = 257524
| Total number of Edges = 0
| Total number of Faces = 29782
| Total number of Cells = 333820
| Sets[4].CGNSType= 20
| Sets[4].IdSet = 4

| BeginSetsIndex = 1 EndSetsIndex = 0
| Issue on writing connectivity -- 3-D
| Invalid element range defined for section 'MATERIAL_SET'

| ----- Original Message -----

| | Hi Carlos,
| 
| | I rebuilt cgns static, and it configured fine;
| 
| | I have though some build errors;
| 
| | CXX WriteSLAC.lo
| 
| | CXX ReadNC.lo
| 
| | CXX ReadGCRM.lo
| 
| | CXX ReadCGNS.lo
| 
| | ../../../moab46/src/io/ReadCGNS.cpp:59:11: warning: unused
| | parameter
| | 'file_set' [-Wunused-parameter]
| 
| | ../../../moab46/src/io/ReadCGNS.cpp:551:11: warning: unused
| | parameter
| | 'element_type' [-Wunused-parameter]
| 
| | ../../../moab46/src/io/ReadCGNS.cpp:551:11: warning: unused
| | parameter
| | 'set_type' [-Wunused-parameter]
| 
| | CXX WriteCGNS.lo
| 
| | In file included from ../../../moab46/src/io/WriteCGNS.cpp:1:0:
| 
| | ../../../moab46/src/io/WriteCGNS.hpp:65:23: error: ISO C++ forbids
| | initialization of member 'IdSet' [-fpermissive]
| 
| | ../../../moab46/src/io/WriteCGNS.hpp:65:23: error: making 'IdSet'
| | static [-fpermissive]
| 
| | ../../../moab46/src/io/WriteCGNS.hpp:65:23: error: ISO C++ forbids
| | in-class initialization of non-const static member 'IdSet'
| 
| | ../../../moab46/src/io/WriteCGNS.hpp:66:22: error: ISO C++ forbids
| | initialization of member 'NbEdges' [-fpermissive]
| 
| | ../../../moab46/src/io/WriteCGNS.hpp:66:22: error: making 'NbEdges'
| | static [-fpermissive]
| 
| | ../../../moab46/src/io/WriteCGNS.hpp:66:22: error: ISO C++ forbids
| | in-class initialization of non-const static member 'NbEdges'
| 
| | ...
| 

| | do you need to initialize these in the struct? I would prefer not
| | to
| | use "-fpermissive"
| 

| | // A structure to store Set information.
| 
| | struct SetStruct {
| 
| | std::string TagName; // Tag name
| 
| | cgsize_t IdSet = -1; // Id of the Set
| 
| | cgsize_t NbEdges=0; // Number of Edges in the Set
| 
| | cgsize_t NbFaces=0; // Number of Faces in the Set
| 
| | cgsize_t NbCells=0; // Number of Cells in the Set
| 
| | // vector with the number of entities in the Sets
| 
| | // 0-MBEDGE | 1-MBTRI | 2-MBQUAD | 3-MBTET | 4-MBPYRAMID |
| | 5-MBPRISM
| | | 6-MBHEX
| 
| | std::vector<cgsize_t> NbEntities;
| 
| | ElementType_t CGNSType;
| 
| | };
| 

| | ----- Original Message -----
| 

| | | Iulian,
| | 
| 

| | | I had a similar issue in the past. I recall that I edited the
| | | cgns.m4
| | | but not exactly which part, probably the section below to link
| | | directly with hdf5_hl, instead of plain hdf5.
| | 
| 

| | | I use Arch linux but not the system libraries. I compile hdf5 and
| | | cgns in a local directory and link those.
| | 
| 

| | | I will take a look into the m4 file and see if I can reproduce
| | | the
| | | problem.
| | 
| 

| | | Regards,
| | 
| 

| | | Carlos
| | 
| 

| | | # Check if cgns is usable by itself
| | 
| 
| | | AC_CHECK_LIB( [cgns], [cg_open], [CGNS_LIBS="-lcgns"], [
| | 
| 
| | | # Check if cgns is usable with HDF5
| | 
| 
| | | unset ac_cv_lib_cgns
| | 
| 
| | | unset ac_cv_lib_cgns_cg_open
| | 
| 
| | | # If we haven't already looked for HDF5 libraries, again now
| | | incase
| | 
| 
| | | # they're in the CGNS lib directory.
| | 
| 
| | | FATHOM_DETECT_HDF5_LIBS
| | 
| 
| | | LDFLAGS="$LDFLAGS $HDF5_LDFLAGS"
| | 
| 
| | | AC_CHECK_LIB( [cgns], [cg_open], [CGNS_LIBS="-lcgns -lhdf5_hl"],
| | | [
| | 
| 
| | | # Try one more time with HDF5 and libcurl
| | 
| 
| | | unset ac_cv_lib_cgns
| | 
| 
| | | unset ac_cv_lib_cgns_cg_open
| | 
| 
| | | AC_CHECK_LIB( [cgns], [cg_open], [CGNS_LIBS="-lcgns -lhdf5_hl
| | | -lcurl"],
| | 
| 
| | | [HAVE_CGNS=no], [-lhdf5_hl $HDF5_LIBS -lcurl] )],
| | 
| 
| | | [-lhdf5_hl $HDF5_LIBS] )],
| | 
| 
| | | )
| | 
| 

| | | On Tue, Nov 19, 2013 at 3:42 PM, Iulian Grindeanu <
| | | iulian at mcs.anl.gov > wrote:
| | 
| 

| | | | Thanks Carlos,
| | | 
| | 
| 
| | | | I have some issues configuring moab with cgns;
| | | 
| | 
| 
| | | | I did not rebuild hdf5, we are using version 1.8.8; cgns seems
| | | | to
| | | | be
| | | | built without problems (I used ccmake)
| | | 
| | 
| 
| | | | But something is wrong with configuring with cgns.
| | | 
| | 
| 
| | | | Do you have any suggestions (besides rebuilding hdf5, cgns,
| | | | ...)
| | | 
| | 
| 

| | | | the symbols needed from hdf5 seem to be there, so I can't
| | | | really
| | | | explain the error. Do you use ubuntu 12 or ubuntu 10? Or
| | | | something
| | | | else?
| | | 
| | 
| 

| | | | /homes/fathom/3rdparty/hdf5-1.8.8-par-mpich2.1.5-gcc/lib> nm
| | | | libhdf5.so | grep H5T_NATIVE_SCHAR_g
| | | 
| | 
| 
| | | | 00000000004cc170 D H5T_NATIVE_SCHAR_g
| | | 
| | 
| 
| | | | jun/homes/fathom/3rdparty/hdf5-1.8.8-par-mpich2.1.5-gcc/lib> nm
| | | | libhdf5.so | grep H5Tget_native_type
| | | 
| | 
| 
| | | | 00000000002424f0 T H5Tget_native_type
| | | 
| | 
| 

| | | | ...
| | | 
| | 
| 
| | | | configure:32106: result: no
| | | 
| | 
| 
| | | | configure:32115: checking for cg_open in -lcgns
| | | 
| | 
| 
| | | | configure:32148:
| | | | /homes/fathom/3rdparty/mpich2/mpich2-1.5/gcc/bin/mpicc -o
| | | | conftest
| | | | -Wall -pipe -pedantic -Wno-long-long -Wextra -Wcast-align
| | | | -Wpointer-arith -Wformat -Wformat-security -Wshadow
| | | | -Wunused-parameter -g -I/homes/fathom/3rdparty/cgns/include
| | | | -DVALGRIND -DUNORDERED_MAP_NS=std::tr1
| | | | -DHAVE_UNORDERED_MAP=tr1/unordered_map
| | | | -DHAVE_UNORDERED_SET=tr1/unordered_set
| | | | -L/homes/fathom/3rdparty/cgns/lib
| | | | -L/homes/fathom/3rdparty/zlib/zlib-1.2.4/gcc/lib
| | | | -L/homes/fathom/3rdparty/szip/szip-2.1/gcc/lib
| | | | -L/homes/fathom/3rdparty/hdf5-1.8.8-par-mpich2.1.5-gcc/lib
| | | | -L/homes/fathom/3rdparty/zlib/zlib-1.2.4/gcc/lib
| | | | -L/homes/fathom/3rdparty/szip/szip-2.1/gcc/lib
| | | | -L/homes/fathom/3rdparty/hdf5-1.8.8-par-mpich2.1.5-gcc/lib
| | | | -L/homes/fathom/3rdparty/zlib/zlib-1.2.4/gcc/lib
| | | | -L/homes/fathom/3rdparty/szip/szip-2.1/gcc/lib
| | | | -L/homes/fathom/3rdparty/hdf5-1.8.8-par-mpich2.1.5-gcc/lib
| | | | -L/homes/fathom/3rdparty/zlib/zlib-1.2.4/gcc/lib
| | | | -L/homes/fathom/3rdparty/szip/szip-2.1/gcc/lib
| | | | -L/homes/fathom/3rdparty/hdf5-1.8.8-par-mpich2.1.5-gcc/lib
| | | | -L/homes/fathom/3rdparty/zlib/zlib-1.2.4/gcc/lib
| | | | -L/homes/fathom/3rdparty/szip/szip-2.1/gcc/lib
| | | | -L/homes/fathom/3rdparty/hdf5-1.8.8-par-mpich2.1.5-gcc/lib
| | | | conftest.c -lcgns -lhdf5_hl -lhdf5 -lcurl -lm
| | | | -L/homes/fathom/3rdparty/mpich2/mpich2-1.5/gcc/lib
| | | | -L/usr/lib/gcc/x86_64-linux-gnu/4.6
| | | | -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu
| | | | -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib
| | | | -L/lib/x86_64-linux-gnu -L/lib/../lib
| | | | -L/usr/lib/x86_64-linux-gnu
| | | | -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../..
| | | | -lmpich -lopa -lmpl -lrt -lpthread -lgfortran -lm -lquadmath
| | | | -L/homes/fathom/3rdparty/mpich2/mpich2-1.5/gcc/lib
| | | | -L/usr/lib/gcc/x86_64-linux-gnu/4.6
| | | | -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu
| | | | -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib
| | | | -L/lib/x86_64-linux-gnu -L/lib/../lib
| | | | -L/usr/lib/x86_64-linux-gnu
| | | | -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../..
| | | | -lmpichf90 -lmpich -lopa -lmpl -lrt -lpthread -lgfortran -lm
| | | | -lquadmath >&5
| | | 
| | 
| 
| | | | /homes/fathom/3rdparty/cgns/lib/libcgns.so: undefined reference
| | | | to
| | | | `
| | | | H5T_NATIVE_SCHAR_g '
| | | 
| | 
| 
| | | | /homes/fathom/3rdparty/cgns/lib/libcgns.so: undefined reference
| | | | to
| | | | `H5Tget_native_type'
| | | 
| | 
| 
| | | | /homes/fathom/3rdparty/cgns/lib/libcgns.so: undefined reference
| | | | to
| | | | `H5Pset_link_creation_order'
| | | 
| | 
| 
| | | | /homes/fathom/3rdparty/cgns/lib/libcgns.so: undefined reference
| | | | to
| | | | `H5P_CLS_DATASET_CREATE_g'
| | | 
| | 
| 
| | | | /homes/fathom/3rdparty/cgns/lib/libcgns.so: undefined reference
| | | | to
| | | | `H5Sget_simple_extent_npoints'
| | | 
| | 
| 
| | | | /homes/fathom/3rdparty/cgns/lib/libcgns.so: undefined reference
| | | | to
| | | | `H5T_IEEE_F64LE_g'
| | | 
| | 
| 
| | | | /homes/fathom/3rdparty/cgns/lib/libcgns.so: undefined reference
| | | | to
| | | | `H5P_CLS_FILE_CREATE_g'
| | | 
| | 
| 
| | | | /homes/fathom/3rdparty/cgns/lib/libcgns.so: undefined reference
| | | | to
| | | | `H5Tcopy'
| | | 
| | 
| 
| | | | /homes/fathom/3rdparty/cgns/lib/libcgns.so: undefined reference
| | | | to
| | | | `H5Fopen'
| | | 
| | 
| 
| | | | ...
| | | 
| | 
| 

| | | | | Iulian, here are some data on the supporting libs:
| | | | 
| | | 
| | 
| 

| | | | | =========
| | | | 
| | | 
| | 
| 

| | | | | HDF5 v1.8.11
| | | | 
| | | 
| | 
| 
| | | | | =========
| | | | 
| | | 
| | 
| 

| | | | | I compile the parallel version, but do not use the parallel
| | | | | IO
| | | | | capabilities yet in CGNS.
| | | | 
| | | 
| | 
| 

| | | | | ./configure \
| | | | 
| | | 
| | 
| 
| | | | | CC=%s \
| | | | 
| | | 
| | 
| 
| | | | | CXX=%s \
| | | | 
| | | 
| | 
| 
| | | | | FC=%s \
| | | | 
| | | 
| | 
| 
| | | | | F9X=%s \
| | | | 
| | | 
| | 
| 
| | | | | RUNPARALLEL=%s \
| | | | 
| | | 
| | 
| 
| | | | | OMPI_MCA_disable_memory_allocator=1 \
| | | | 
| | | 
| | 
| 
| | | | | --prefix=%s \
| | | | 
| | | 
| | 
| 
| | | | | --enable-largefile \
| | | | 
| | | 
| | 
| 
| | | | | --enable-unsupported \
| | | | 
| | | 
| | 
| 
| | | | | --enable-shared \
| | | | 
| | | 
| | 
| 
| | | | | --disable-static \
| | | | 
| | | 
| | 
| 
| | | | | --enable-production=yes \
| | | | 
| | | 
| | 
| 
| | | | | --with-pthread=%s,%s \
| | | | 
| | | 
| | 
| 
| | | | | --with-zlib=%s,%s \
| | | | 
| | | 
| | 
| 
| | | | | --with-default-api-version=v18 \
| | | | 
| | | 
| | 
| 
| | | | | --enable-parallel=yes \
| | | | 
| | | 
| | 
| 
| | | | | --enable-cxx \
| | | | 
| | | 
| | 
| 
| | | | | --disable-sharedlib-rpath"
| | | | 
| | | 
| | 
| 

| | | | | hdf5_config.log attached
| | | | 
| | | 
| | 
| 

| | | | | =========
| | | | 
| | | 
| | 
| 
| | | | | CGNS v3.2.1 (Beta)
| | | | 
| | | 
| | 
| 
| | | | | =========
| | | | 
| | | 
| | 
| 

| | | | | We have not considered the parallel IO capabilities as of
| | | | | now.
| | | | | There
| | | | | is no clear definition yet how to handle MIXED type element
| | | | | meshes,
| | | | | for which you can not specify a fixed record lenght to
| | | | | retrieve
| | | | | with
| | | | | hdf5. Although the HDF5 library I linked is mpi-enabled.
| | | | 
| | | 
| | 
| 

| | | | | Junior at some point tested with version 3.1.4 release 2 and
| | | | | it
| | | | | worked fine, AFAIK.
| | | | 
| | | 
| | 
| 

| | | | | The library is transitioning to CMake, I've used:
| | | | 
| | | 
| | 
| 

| | | | | cmake \
| | | | 
| | | 
| | 
| 
| | | | | -DCGNS_BUILD_SHARED:BOOL=ON \
| | | | 
| | | 
| | 
| 
| | | | | -DCMAKE_BUILD_TYPE:STRING=Release \
| | | | 
| | | 
| | 
| 
| | | | | -DCMAKE_INSTALL_PREFIX:PATH=%s \
| | | | 
| | | 
| | 
| 
| | | | | -DCMAKE_SKIP_RPATH:BOOL=ON \
| | | | 
| | | 
| | 
| 
| | | | | -DCGNS_BUILD_CGNSTOOLS:BOOL=OFF \
| | | | 
| | | 
| | 
| 
| | | | | -DCGNS_ENABLE_64BIT:BOOL=ON \
| | | | 
| | | 
| | 
| 
| | | | | -DCGNS_ENABLE_FORTRAN:BOOL=OFF \
| | | | 
| | | 
| | 
| 
| | | | | -DCGNS_ENABLE_PARALLEL:BOOL=ON \
| | | | 
| | | 
| | 
| 
| | | | | -DCGNS_ENABLE_HDF5:BOOL=ON \
| | | | 
| | | 
| | 
| 
| | | | | -DHDF5_INCLUDE_PATH:PATH=%s \
| | | | 
| | | 
| | 
| 
| | | | | -DHDF5_LIBRARY:FILEPATH=%s \
| | | | 
| | | 
| | 
| 
| | | | | -DHDF5_NEED_MPI:BOOL=ON \
| | | | 
| | | 
| | 
| 
| | | | | -DHDF5_NEED_ZLIB:BOOL=ON \
| | | | 
| | | 
| | 
| 
| | | | | -DZLIB_LIBRARY:FILEPATH=%s \
| | | | 
| | | 
| | 
| 
| | | | | -DCGNS_ENABLE_SCOPING:BOOL=OFF \
| | | | 
| | | 
| | 
| 
| | | | | -DCGNS_ENABLE_TESTS:BOOL=OFF \
| | | | 
| | | 
| | 
| 
| | | | | -DMPIEXEC:FILEPATH=%s \
| | | | 
| | | 
| | 
| 
| | | | | -DMPI_C_COMPILER:FILEPATH=%s \
| | | | 
| | | 
| | 
| 
| | | | | -DMPI_C_INCLUDE_PATH:PATH=\"%s\"
| | | | 
| | | 
| | 
| 

| | | | | cgns321_CMakeCache.txt is attached.
| | | | 
| | | 
| | 
| 
| | | | | =========
| | | | 
| | | 
| | 
| 
| | | | | MOAB v4.6.2
| | | | 
| | | 
| | 
| 
| | | | | =========
| | | | 
| | | 
| | 
| 

| | | | | The patch is against the tar file from the website (
| | | | | http://ftp.mcs.anl.gov/pub/fathom/moab-4.6.2.tar.gz ). There
| | | | | is
| | | | | no
| | | | | need from our side to keep it on 462. Feel free to include it
| | | | | in
| | | | | a
| | | | | latter release. It was only convenient for me to keep the
| | | | | patch
| | | | | against a fixed release. For this, I packaged in the previous
| | | | | email
| | | | | a tar.gz with the modified and new sources and sample cgns
| | | | | files.
| | | | 
| | | 
| | 
| 

| | | | | The reader and writer are serial only. We tested against
| | | | | homogeneous
| | | | | and heterogeneous meshes up to 20million elements and it
| | | | | worked.
| | | | | You
| | | | | can read a CGNS file, convert it to H5M and then convert it
| | | | | back
| | | | | to
| | | | | CGNS.
| | | | 
| | | 
| | 
| 

| | | | | The modifications to compile moab is just an --with-cgns=DIR
| | | | | flag.
| | | | 
| | | 
| | 
| 

| | | | | These build use shared libs for now, as I am working on my
| | | | | workstation. Latter I will experiment with static objects for
| | | | | clusters.
| | | | 
| | | 
| | 
| 

| | | | | If you need more details, let us know.
| | | | 
| | | 
| | 
| 

| | | | | Regards,
| | | | 
| | | 
| | 
| 
| | | | | Carlos
| | | | 
| | | 
| | 
| 

| | | | | On Tue, Nov 19, 2013 at 1:30 PM, Iulian Grindeanu <
| | | | | iulian at mcs.anl.gov > wrote:
| | | | 
| | | 
| | 
| 

| | | | | | Thank you Carlos and Junior, for your help.
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | I will need to build cgns library; you mentioned that you
| | | | | | tested
| | | | | | 3.2
| | | | | | beta, too; I will probably stick with 3.1.4, (release 2?).
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | Also, what version of hdf5 library do you use? Should it
| | | | | | work
| | | | | | for
| | | | | | any
| | | | | | version greater than 1.8?
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | I assume you build statically? Can I also get your
| | | | | | config.log
| | | | | | files
| | | | | | for cgns and moab?
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | I assume that you are not using any of the parallel io
| | | | | | capability,
| | | | | | because that seems to be available only in beta version.
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | So everything is read/written in serial, isn't it?
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | I am not sure if we will port the cgns reader / writer to
| | | | | | 4.6.2
| | | | | | also.
| | | | | | Is your patch against 4.6.2 tar file or against
| | | | | | "Version4.6"
| | | | | | branch?
| | | | | | There should not be much difference.
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | Tim, what do you suggest? In a way, it should be easier to
| | | | | | update
| | | | | | 4.6.2. But I don't want to merge to the master after that,
| | | | | | there
| | | | | | are
| | | | | | too many changes. This patch should work out of the box for
| | | | | | 4.6.2,
| | | | | | after I build cgns.
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | Best Regards,
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | Iulian
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | Iulian,
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | Sorry for the delay in getting back to you... I have
| | | | | | | tested
| | | | | | | the
| | | | | | | new
| | | | | | | version and it worked OK for my mesh sample. I will test
| | | | | | | it
| | | | | | | latter
| | | | | | | for other cases.
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | Also, I am sending you the CGNS reader and writer along
| | | | | | | with
| | | | | | | input
| | | | | | | cgns mesh examples. We have worked on top of version 462,
| | | | | | | so
| | | | | | | things
| | | | | | | may look a little out of place from the repository.
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | The attached patch modifies some autotools files and new
| | | | | | | files
| | | | | | | (apply
| | | | | | | with patch -p0 -i moab462_cgns.patch). Moreover, the
| | | | | | | tar.gz
| | | | | | | contains
| | | | | | | sample meshes (simple airfoil mesh in 2D and a mixed-mesh
| | | | | | | discretization of a 3D wingbody geometry).
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | We have tested the CGNS capability with the latest
| | | | | | | library
| | | | | | | vesion
| | | | | | | 3.1.4 and 3.2-beta. Furthermore, we sticked with the base
| | | | | | | CGNS
| | | | | | | use-case, that is, single base/zone. We will also
| | | | | | | implement
| | | | | | | latter
| | | | | | | capability to write out user variables.
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | Please, feel free to improve the files and if you have
| | | | | | | any
| | | | | | | questions,
| | | | | | | let us know. carbrevi at gmail.com and junior.hmg at gmail.com
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | Regards,
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | Carlos
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | On Sat, Nov 16, 2013 at 3:57 AM, Iulian Grindeanu <
| | | | | | | iulian at mcs.anl.gov > wrote:
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | Hi Carlos,
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | That ticket is now closed, Tim fixed the bug.
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | Please let us know if it works for you.
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | Thanks for your patience,
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | Iulian
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | Iulian,
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | thanks for checking this out. I will be following the
| | | | | | | | | trac
| | | | | | | | | ticket.
| | | | | | | | | The mesh I used is attached, partitioned for 8 procs
| | | | | | | | | (all
| | | | | | | | | tri).
| | | | | | | | | I
| | | | | | | | | noticed that are much simpler sample meshes you
| | | | | | | | | created
| | | | | | | | | to
| | | | | | | | | investigate this issue. Anyway, you can use the
| | | | | | | | | airfoil
| | | | | | | | | mesh
| | | | | | | | | if
| | | | | | | | | you
| | | | | | | | | find it useful.
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | I use a separate code to compute elements adjacency
| | | | | | | | | and
| | | | | | | | | then
| | | | | | | | | pass
| | | | | | | | | those to metis. To write the partitions into h5m I
| | | | | | | | | followed
| | | | | | | | | the
| | | | | | | | | code
| | | | | | | | | form mbzoltan tool. The partitioning code is correct
| | | | | | | | | (element-based)
| | | | | | | | | since I have used this code previously with other
| | | | | | | | | applications...
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | I am from Brazil indeed, utc -3h. I'm a PhD candidate
| | | | | | | | | from
| | | | | | | | | Technological Institute of Aeronautics in Sao Jose
| | | | | | | | | dos
| | | | | | | | | Campos.
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | Now I am building/adapting parts of my research code
| | | | | | | | | (high-order
| | | | | | | | | unstructured CFD) to use MOAB. I will stick with 462
| | | | | | | | | for
| | | | | | | | | this,
| | | | | | | | | working with non-mixed meshes for this transition
| | | | | | | | | period.
| | | | | | | | | In
| | | | | | | | | the
| | | | | | | | | near feature I will also look into the structured
| | | | | | | | | meshes
| | | | | | | | | capability
| | | | | | | | | of the library, as well as the high-order elements
| | | | | | | | | implementation.
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | The CGNS reader/writer is in the works too and I will
| | | | | | | | | submit
| | | | | | | | | those
| | | | | | | | | as
| | | | | | | | | well. My co-worker Junior, cc'ed here, is dealing
| | | | | | | | | with
| | | | | | | | | the
| | | | | | | | | writer
| | | | | | | | | part and we are almos done.
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | Regards,
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | Carlos Breviglieri
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | On Sat, Oct 12, 2013 at 5:40 PM, Iulian Grindeanu <
| | | | | | | | | iulian at mcs.anl.gov > wrote:
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | Hi Carlos,
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | I am seeing a problem with ghosting, thank you for
| | | | | | | | | | pointing
| | | | | | | | | | it
| | | | | | | | | | out.
| | | | | | | | | | Danqing and I messed up with that code :( to fix
| | | | | | | | | | some
| | | | | | | | | | other
| | | | | | | | | | issues
| | | | | | | | | | we were seeing.
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | http://trac.mcs.anl.gov/projects/ITAPS/ticket/284
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | It may take a while to fix it properly.
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | Thanks again,
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | Iulian
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | Hmmmm,
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | Can you send me the 2d_naca0012.h5m with your
| | | | | | | | | | | partition?
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | If you do ghosting after reading, did you get the
| | | | | | | | | | | same
| | | | | | | | | | | results?
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | The elements/processors should not change
| | | | | | | | | | | ownership
| | | | | | | | | | | after
| | | | | | | | | | | ghosting,
| | | | | | | | | | | but maybe they do.
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | It is indeed a work in progress, you may have
| | | | | | | | | | | found
| | | | | | | | | | | another
| | | | | | | | | | | issue
| | | | | | | | | | | :(
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | Not being able to write is probably the biggest
| | | | | | | | | | | problem.
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | On what time zone do you work? :) Also, where are
| | | | | | | | | | | you
| | | | | | | | | | | from?
| | | | | | | | | | | The
| | | | | | | | | | | name
| | | | | | | | | | | looks Brazilian, Portuguese, or Italian ? Or
| | | | | | | | | | | Spanish?
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | Thanks,
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | Iulian
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | Hi Iulian,
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | I have just ran some tests with the latest
| | | | | | | | | | | | clone
| | | | | | | | | | | | of
| | | | | | | | | | | | the
| | | | | | | | | | | | master
| | | | | | | | | | | | repo
| | | | | | | | | | | | (saturday morning). Here are my findings with
| | | | | | | | | | | | MOAB
| | | | | | | | | | | | 470pre:
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | Now I am able to read partitined mixed meshes
| | | | | | | | | | | | without
| | | | | | | | | | | | errors.
| | | | | | | | | | | | However, if I plot the owned entitied for a
| | | | | | | | | | | | given
| | | | | | | | | | | | proc,
| | | | | | | | | | | | even
| | | | | | | | | | | | for
| | | | | | | | | | | | non-mixed meshes, they are different from the
| | | | | | | | | | | | ones
| | | | | | | | | | | | obtained
| | | | | | | | | | | | by
| | | | | | | | | | | | the
| | | | | | | | | | | | partitioner, see below
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | Mesh distribution for 2d_naca0012.h5m (all tri
| | | | | | | | | | | | mesh)
| | | | | | | | | | | | from
| | | | | | | | | | | | partitioner, over 8 procs:
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | proc[0] has 866 elements
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | proc[1] has 863 elements
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | proc[2] has 866 elements
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | proc[3] has 869 elements
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | proc[4] has 872 elements
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | proc[5] has 869 elements
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | proc[6] has 862 elements
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | proc[7] has 877 elements
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | Mesh distribution seen by MOAB 470pre
| | | | | | | | | | | | ("PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS;PARALLEL_GHOSTS=2.0.1")
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | owned_entities[5], size = 864
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | owned_entities[6], size = 853
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | owned_entities[7], size = 871
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | owned_entities[1], size = 859
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | owned_entities[3], size = 860
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | owned_entities[4], size = 867
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | owned_entities[0], size = 866
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | owned_entities[2], size = 861
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | besides procs 0, all other report wrong number
| | | | | | | | | | | | of
| | | | | | | | | | | | entities.
| | | | | | | | | | | | Code
| | | | | | | | | | | | to
| | | | | | | | | | | | compute such distribution is below (based on
| | | | | | | | | | | | example/HelloParMOAB.cpp). The mixed mesh
| | | | | | | | | | | | (2d_naca0012_mixed.h5m)
| | | | | | | | | | | | now is read (does not crash) but the
| | | | | | | | | | | | distribution
| | | | | | | | | | | | is
| | | | | | | | | | | | off
| | | | | | | | | | | | as
| | | | | | | | | | | | well.
| | | | | | | | | | | | With MOAB 462 the distribution is OK for
| | | | | | | | | | | | homogeneous
| | | | | | | | | | | | meshes.
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | Moreover, with MOAB 470pre, no output is
| | | | | | | | | | | | written
| | | | | | | | | | | | to
| | | | | | | | | | | | disk
| | | | | | | | | | | | with
| | | | | | | | | | | | PARALLEL=WRITE_PART, regardless of the mesh
| | | | | | | | | | | | type.
| | | | | | | | | | | | Using
| | | | | | | | | | | | PARALLEL=NONE works, but only one part of the
| | | | | | | | | | | | domain
| | | | | | | | | | | | is
| | | | | | | | | | | | written,
| | | | | | | | | | | | as
| | | | | | | | | | | | expected.
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | I understand that this is a work in progress.
| | | | | | | | | | | | If
| | | | | | | | | | | | you
| | | | | | | | | | | | need
| | | | | | | | | | | | more
| | | | | | | | | | | | information, let me know.
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | Regards,
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | Carlos Breviglieri
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | read_options =
| | | | | | | | | | | | "PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS;PARALLEL_GHOSTS=2.0.1";
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | moab::Interface* mb = new Core;
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | // Create root sets for each mesh. Then pass
| | | | | | | | | | | | these
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | // to the load_file functions to be populated.
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | EntityHandle rootset, partnset;
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | mb->create_meshset(MESHSET_SET, rootset);
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | mb->create_meshset(MESHSET_SET, partnset);
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | // Create the parallel communicator object with
| | | | | | | | | | | | the
| | | | | | | | | | | | partition
| | | | | | | | | | | | handle
| | | | | | | | | | | | associated with MOAB
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | ParallelComm *pcomm =
| | | | | | | | | | | | ParallelComm::get_pcomm(mb,
| | | | | | | | | | | | partnset,
| | | | | | | | | | | | &myComm);
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | // Load the file from disk with given options
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | mb->load_file(meshFile.c_str(), &rootset,
| | | | | | | | | | | | read_options.c_str());
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | // Get all entities of dimension = dim
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | Range elemRange, owned_entities;
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | int dim = 2;
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | mb->get_entities_by_dimension(rootset, dim,
| | | | | | | | | | | | elemRange,
| | | | | | | | | | | | false);
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | pcomm->filter_pstatus(elemRange, // pass
| | | | | | | | | | | | entities
| | | | | | | | | | | | that
| | | | | | | | | | | | we
| | | | | | | | | | | | want
| | | | | | | | | | | | to
| | | | | | | | | | | | filter
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | PSTATUS_NOT_OWNED, // status we are looking for
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | PSTATUS_NOT, // operation applied ; so it will
| | | | | | | | | | | | return
| | | | | | | | | | | | owned
| | | | | | | | | | | | entities
| | | | | | | | | | | | (!not_owned = owned)
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | -1, // this means all processors
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | &owned_entities);
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | std::vector<int> procID(owned_entities.size(),
| | | | | | | | | | | | myRank);
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | std::cout << "owned_entities[" << myRank << "],
| | | | | | | | | | | | size
| | | | | | | | | | | | =
| | | | | | | | | | | | "
| | | | | | | | | | | | <<
| | | | | | | | | | | | owned_entities.size() << std::endl;
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | Tag procID_tag;
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | mb->tag_get_handle("PROC_ID", 1,
| | | | | | | | | | | | MB_TYPE_INTEGER,
| | | | | | | | | | | | procID_tag,
| | | | | | | | | | | | MB_TAG_CREAT | MB_TAG_DENSE, &procID[0]);
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | mb->tag_set_data(procID_tag, owned_entities,
| | | | | | | | | | | | &procID[0]);
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | // WRITE_PART writes all partitions to a single
| | | | | | | | | | | | output
| | | | | | | | | | | | file
| | | | | | | | | | | | (only
| | | | | | | | | | | | h5m
| | | | | | | | | | | | format supports parallel IO at the moment).
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | // One can use the mbconvert tool to convert
| | | | | | | | | | | | the
| | | | | | | | | | | | output
| | | | | | | | | | | | to
| | | | | | | | | | | | other
| | | | | | | | | | | | formats.
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | mb->write_file(outputFile.c_str(), "H5M",
| | | | | | | | | | | | "PARALLEL=WRITE_PART");
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | On Thu, Oct 10, 2013 at 10:37 AM, Tim Tautges <
| | | | | | | | | | | | tautges at mcs.anl.gov
| | | | | | | | | | | | >
| | | | | | | | | | | | wrote:
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | Yeah, too complicated to backport, and latest
| | | | | | | | | | | | | works
| | | | | | | | | | | | | for
| | | | | | | | | | | | | Carlos
| | | | | | | | | | | | | anyway.
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | - tim
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | On 10/09/2013 09:41 PM, Iulian Grindeanu
| | | | | | | | | | | | | wrote:
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | | ------------------------------
| | | | | | | | | | | | | | ------------------------------
| | | | | | | | | | | | | | ------------------------------
| | | | | | | | | | | | | | ------------------------------
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | | Both worked for me, with the current code
| | | | | | | | | | | | | | (4.7.0pre)
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | | I don't get your error :(
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | What version are you using? I will try
| | | | | | | | | | | | | | 4.6.2,
| | | | | | | | | | | | | | but
| | | | | | | | | | | | | | it
| | | | | | | | | | | | | | should
| | | | | | | | | | | | | | be
| | | | | | | | | | | | | | fine
| | | | | | | | | | | | | | there too :(
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | | OK, I got an error with more quads, on
| | | | | | | | | | | | | | 4.6.2
| | | | | | | | | | | | | | maybe
| | | | | | | | | | | | | | I
| | | | | | | | | | | | | | mixed
| | | | | | | | | | | | | | them
| | | | | | | | | | | | | | up
| | | | | | | | | | | | | | when I saved :(
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | mpiexec -np 2 /home/iulian/source/MOAB46/
| | | | | | | | | | | | | | tools/mbconvert
| | | | | | | | | | | | | | -O
| | | | | | | | | | | | | | PARALLEL=READ_PART -O
| | | | | | | | | | | | | | PARTITION=PARALLEL_PARTITION
| | | | | | | | | | | | | | -O
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | PARALLEL_RESOLVE_SHARED_ENTS -O
| | | | | | | | | | | | | | PARALLEL_GHOSTS=2.0.1
| | | | | | | | | | | | | | -o
| | | | | | | | | | | | | | PARALLEL=WRITE_PART
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | /home/iulian/tmp/2d_naca0012_ mixed2.h5m
| | | | | | | | | | | | | | 2.h5m
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Leaked HDF5 object handle in function at
| | | | | | | | | | | | | | ../../../moab46source/src/io/
| | | | | | | | | | | | | | ReadHDF5.cpp:1523
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Open at entrance: 1
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Open at exit: 2
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Leaked HDF5 object handle in function at
| | | | | | | | | | | | | | ../../../moab46source/src/io/
| | | | | | | | | | | | | | ReadHDF5.cpp:827
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Open at entrance: 1
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Open at exit: 2
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Failed to load
| | | | | | | | | | | | | | "/home/iulian/tmp/2d_naca0012_
| | | | | | | | | | | | | | mixed2.h5m".
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Error code: MB_INDEX_OUT_OF_RANGE (1)
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Error message: Failed in step PARALLEL READ
| | | | | | | | | | | | | | PART
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Cannot close file with open handles: 0
| | | | | | | | | | | | | | file,
| | | | | | | | | | | | | | 1
| | | | | | | | | | | | | | data,
| | | | | | | | | | | | | | 0
| | | | | | | | | | | | | | group,
| | | | | | | | | | | | | | 0
| | | | | | | | | | | | | | type,
| | | | | | | | | | | | | | 0 attr
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | | I will look into it.
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | | Hi Carlos,
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | It looks like it is a bug in 4.6.2.
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | I don't know if it will be fixed, there are
| | | | | | | | | | | | | | some
| | | | | | | | | | | | | | important
| | | | | | | | | | | | | | changes
| | | | | | | | | | | | | | in
| | | | | | | | | | | | | | ghosting in current version.
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | So for Version4.6 branch, the model with 17
| | | | | | | | | | | | | | quads
| | | | | | | | | | | | | | works
| | | | | | | | | | | | | | fine
| | | | | | | | | | | | | | if
| | | | | | | | | | | | | | you
| | | | | | | | | | | | | | don't do ghosting:
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | | iulian at T520-iuli:~/source/ MOAB46$ mpiexec
| | | | | | | | | | | | | | -np
| | | | | | | | | | | | | | 2
| | | | | | | | | | | | | | /home/iulian/source/MOAB46/ tools/mbconvert
| | | | | | | | | | | | | | -O
| | | | | | | | | | | | | | PARALLEL=READ_PART
| | | | | | | | | | | | | | -O
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | PARTITION=PARALLEL_PARTITION -O
| | | | | | | | | | | | | | PARALLEL_RESOLVE_SHARED_ENTS
| | | | | | | | | | | | | | -o
| | | | | | | | | | | | | | PARALLEL=WRITE_PART
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | /home/iulian/tmp/2d_naca0012_
| | | | | | | | | | | | | | mixed_invert.h5m
| | | | | | | | | | | | | | 2.h5m
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Read "/home/iulian/tmp/2d_naca0012_
| | | | | | | | | | | | | | mixed_invert.h5m"
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Wrote "2.h5m"
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | | I would recommend upgrading to current
| | | | | | | | | | | | | | version.
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | That code is pretty complicated, and I am
| | | | | | | | | | | | | | not
| | | | | | | | | | | | | | sure
| | | | | | | | | | | | | | if
| | | | | | | | | | | | | | we
| | | | | | | | | | | | | | will
| | | | | | | | | | | | | | backport changes to Version4.6 branch.
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | | Tim, what do you suggest? Should I try to
| | | | | | | | | | | | | | backport
| | | | | | | | | | | | | | some
| | | | | | | | | | | | | | changes
| | | | | | | | | | | | | | in
| | | | | | | | | | | | | | ParallelComm? I know you are working on
| | | | | | | | | | | | | | that
| | | | | | | | | | | | | | code.
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | | Thanks,
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | | Iulian
| | | | | | | | | | | | | 
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | --
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | ==============================
| | | | | | | | | | | | | ==============================
| | | | | | | | | | | | | ====
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | "You will keep in perfect peace him whose
| | | | | | | | | | | | | mind
| | | | | | | | | | | | | is
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | steadfast, because he trusts in you." Isaiah
| | | | | | | | | | | | | 26:3
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 

| | | | | | | | | | | | | Tim Tautges Argonne National Laboratory
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | ( tautges at mcs.anl.gov ) (telecommuting from
| | | | | | | | | | | | | UW-Madison)
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | phone (gvoice): (608) 354-1459 1500
| | | | | | | | | | | | | Engineering
| | | | | | | | | | | | | Dr.
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
| | | | | | | | | | | | | fax: (608) 263-4499 Madison, WI 53706
| | | | | | | | | | | | 
| | | | | | | | | | | 
| | | | | | | | | | 
| | | | | | | | | 
| | | | | | | | 
| | | | | | | 
| | | | | | 
| | | | | 
| | | | 
| | | 
| | 
| 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/moab-dev/attachments/20131119/9e4ea74b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch_cgnswriter
Type: text/x-patch
Size: 1456 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/moab-dev/attachments/20131119/9e4ea74b/attachment-0001.bin>


More information about the moab-dev mailing list