[MOAB-dev] mixed mesh question
Carlos Breviglieri
carbrevi at gmail.com
Sat Oct 12 06:00:43 CDT 2013
Hi Iulian,
I have just ran some tests with the latest clone of the master repo
(saturday morning). Here are my findings with MOAB 470pre:
Now I am able to read partitined mixed meshes without errors. However, if I
plot the owned entitied for a given proc, even for non-mixed meshes, they
are different from the ones obtained by the partitioner, see below
Mesh distribution for 2d_naca0012.h5m (all tri mesh) from partitioner, over
8 procs:
proc[0] has 866 elements
proc[1] has 863 elements
proc[2] has 866 elements
proc[3] has 869 elements
proc[4] has 872 elements
proc[5] has 869 elements
proc[6] has 862 elements
proc[7] has 877 elements
Mesh distribution seen by MOAB 470pre
("PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS;PARALLEL_GHOSTS=2.0.1")
owned_entities[5], size = 864
owned_entities[6], size = 853
owned_entities[7], size = 871
owned_entities[1], size = 859
owned_entities[3], size = 860
owned_entities[4], size = 867
owned_entities[0], size = 866
owned_entities[2], size = 861
besides procs 0, all other report wrong number of entities. Code to compute
such distribution is below (based on example/HelloParMOAB.cpp). The mixed
mesh (2d_naca0012_mixed.h5m) now is read (does not crash) but the
distribution is off as well. With MOAB 462 the distribution is OK for
homogeneous meshes.
Moreover, with MOAB 470pre, no output is written to disk with
PARALLEL=WRITE_PART, regardless of the mesh type. Using PARALLEL=NONE
works, but only one part of the domain is written, as expected.
I understand that this is a work in progress. If you need more information,
let me know.
Regards,
Carlos Breviglieri
read_options =
"PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS;PARALLEL_GHOSTS=2.0.1";
moab::Interface* mb = new Core;
// Create root sets for each mesh. Then pass these
// to the load_file functions to be populated.
EntityHandle rootset, partnset;
mb->create_meshset(MESHSET_SET, rootset);
mb->create_meshset(MESHSET_SET, partnset);
// Create the parallel communicator object with the partition handle
associated with MOAB
ParallelComm *pcomm = ParallelComm::get_pcomm(mb, partnset, &myComm);
// Load the file from disk with given options
mb->load_file(meshFile.c_str(), &rootset, read_options.c_str());
// Get all entities of dimension = dim
Range elemRange, owned_entities;
int dim = 2;
mb->get_entities_by_dimension(rootset, dim, elemRange, false);
pcomm->filter_pstatus(elemRange, // pass entities that we want to filter
PSTATUS_NOT_OWNED, // status we are looking for
PSTATUS_NOT, // operation applied ; so it will
return owned entities (!not_owned = owned)
-1, // this means all processors
&owned_entities);
std::vector<int> procID(owned_entities.size(), myRank);
std::cout << "owned_entities[" << myRank << "], size = " <<
owned_entities.size() << std::endl;
Tag procID_tag;
mb->tag_get_handle("PROC_ID", 1, MB_TYPE_INTEGER, procID_tag,
MB_TAG_CREAT | MB_TAG_DENSE, &procID[0]);
mb->tag_set_data(procID_tag, owned_entities, &procID[0]);
// WRITE_PART writes all partitions to a single output file (only h5m
format supports parallel IO at the moment).
// One can use the mbconvert tool to convert the output to other
formats.
mb->write_file(outputFile.c_str(), "H5M", "PARALLEL=WRITE_PART");
On Thu, Oct 10, 2013 at 10:37 AM, Tim Tautges <tautges at mcs.anl.gov> wrote:
> Yeah, too complicated to backport, and latest works for Carlos anyway.
>
> - tim
>
> On 10/09/2013 09:41 PM, Iulian Grindeanu wrote:
>
>>
>>
>> ------------------------------**------------------------------**
>> ------------------------------**------------------------------
>>
>>
>> Both worked for me, with the current code (4.7.0pre)
>>
>> I don't get your error :(
>> What version are you using? I will try 4.6.2, but it should be fine
>> there too :(
>>
>>
>> OK, I got an error with more quads, on 4.6.2 maybe I mixed them up
>> when I saved :(
>> mpiexec -np 2 /home/iulian/source/MOAB46/**tools/mbconvert -O
>> PARALLEL=READ_PART -O PARTITION=PARALLEL_PARTITION -O
>> PARALLEL_RESOLVE_SHARED_ENTS -O PARALLEL_GHOSTS=2.0.1 -o
>> PARALLEL=WRITE_PART
>> /home/iulian/tmp/2d_naca0012_**mixed2.h5m 2.h5m
>> Leaked HDF5 object handle in function at ../../../moab46source/src/io/
>> **ReadHDF5.cpp:1523
>> Open at entrance: 1
>> Open at exit: 2
>> Leaked HDF5 object handle in function at ../../../moab46source/src/io/
>> **ReadHDF5.cpp:827
>> Open at entrance: 1
>> Open at exit: 2
>> Failed to load "/home/iulian/tmp/2d_naca0012_**mixed2.h5m".
>> Error code: MB_INDEX_OUT_OF_RANGE (1)
>> Error message: Failed in step PARALLEL READ PART
>> Cannot close file with open handles: 0 file, 1 data, 0 group, 0 type,
>> 0 attr
>>
>>
>> I will look into it.
>>
>> Hi Carlos,
>> It looks like it is a bug in 4.6.2.
>> I don't know if it will be fixed, there are some important changes in
>> ghosting in current version.
>> So for Version4.6 branch, the model with 17 quads works fine if you don't
>> do ghosting:
>>
>> iulian at T520-iuli:~/source/**MOAB46$ mpiexec -np 2
>> /home/iulian/source/MOAB46/**tools/mbconvert -O PARALLEL=READ_PART -O
>> PARTITION=PARALLEL_PARTITION -O PARALLEL_RESOLVE_SHARED_ENTS -o
>> PARALLEL=WRITE_PART
>> /home/iulian/tmp/2d_naca0012_**mixed_invert.h5m 2.h5m
>> Read "/home/iulian/tmp/2d_naca0012_**mixed_invert.h5m"
>> Wrote "2.h5m"
>>
>> I would recommend upgrading to current version.
>> That code is pretty complicated, and I am not sure if we will backport
>> changes to Version4.6 branch.
>>
>> Tim, what do you suggest? Should I try to backport some changes in
>> ParallelComm? I know you are working on that code.
>>
>> Thanks,
>> Iulian
>>
>>
>>
>>
>>
>>
> --
> ==============================**==============================**====
> "You will keep in perfect peace him whose mind is
> steadfast, because he trusts in you." Isaiah 26:3
>
> Tim Tautges Argonne National Laboratory
> (tautges at mcs.anl.gov) (telecommuting from UW-Madison)
> phone (gvoice): (608) 354-1459 1500 Engineering Dr.
> fax: (608) 263-4499 Madison, WI 53706
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/moab-dev/attachments/20131012/870a12c1/attachment.html>
More information about the moab-dev
mailing list