[MOAB-dev] mixed mesh question

Sat Oct 12 15:40:42 CDT 2013

Hi Carlos, 
I am seeing a problem with ghosting, thank you for pointing it out. Danqing and I messed up with that code :( to fix some other issues we were seeing. 
http://trac.mcs.anl.gov/projects/ITAPS/ticket/284 

It may take a while to fix it properly. 
Thanks again, 
Iulian 

----- Original Message -----

| Hmmmm,
| Can you send me the 2d_naca0012.h5m with your partition?
| If you do ghosting after reading, did you get the same results?
| The elements/processors should not change ownership after ghosting,
| but maybe they do.
| It is indeed a work in progress, you may have found another issue :(
| Not being able to write is probably the biggest problem.

| On what time zone do you work? :) Also, where are you from? The name
| looks Brazilian, Portuguese, or Italian ? Or Spanish?

| Thanks,
| Iulian

| ----- Original Message -----

| | Hi Iulian,
| 

| | I have just ran some tests with the latest clone of the master repo
| | (saturday morning). Here are my findings with MOAB 470pre:
| 

| | Now I am able to read partitined mixed meshes without errors.
| | However, if I plot the owned entitied for a given proc, even for
| | non-mixed meshes, they are different from the ones obtained by the
| | partitioner, see below
| 

| | Mesh distribution for 2d_naca0012.h5m (all tri mesh) from
| | partitioner, over 8 procs:
| 
| | proc[0] has 866 elements
| 
| | proc[1] has 863 elements
| 
| | proc[2] has 866 elements
| 
| | proc[3] has 869 elements
| 
| | proc[4] has 872 elements
| 
| | proc[5] has 869 elements
| 
| | proc[6] has 862 elements
| 
| | proc[7] has 877 elements
| 

| | Mesh distribution seen by MOAB 470pre
| | ("PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS;PARALLEL_GHOSTS=2.0.1")
| 
| | owned_entities[5], size = 864
| 
| | owned_entities[6], size = 853
| 
| | owned_entities[7], size = 871
| 
| | owned_entities[1], size = 859
| 
| | owned_entities[3], size = 860
| 
| | owned_entities[4], size = 867
| 
| | owned_entities[0], size = 866
| 
| | owned_entities[2], size = 861
| 

| | besides procs 0, all other report wrong number of entities. Code to
| | compute such distribution is below (based on
| | example/HelloParMOAB.cpp). The mixed mesh (2d_naca0012_mixed.h5m)
| | now is read (does not crash) but the distribution is off as well.
| | With MOAB 462 the distribution is OK for homogeneous meshes.
| 

| | Moreover, with MOAB 470pre, no output is written to disk with
| | PARALLEL=WRITE_PART, regardless of the mesh type. Using
| | PARALLEL=NONE works, but only one part of the domain is written, as
| | expected.
| 

| | I understand that this is a work in progress. If you need more
| | information, let me know.
| 

| | Regards,
| 

| | Carlos Breviglieri
| 

| | read_options =
| | "PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS;PARALLEL_GHOSTS=2.0.1";
| 

| | moab::Interface* mb = new Core;
| 

| | // Create root sets for each mesh. Then pass these
| 
| | // to the load_file functions to be populated.
| 
| | EntityHandle rootset, partnset;
| 
| | mb->create_meshset(MESHSET_SET, rootset);
| 
| | mb->create_meshset(MESHSET_SET, partnset);
| 

| | // Create the parallel communicator object with the partition
| | handle
| | associated with MOAB
| 
| | ParallelComm *pcomm = ParallelComm::get_pcomm(mb, partnset,
| | &myComm);
| 

| | // Load the file from disk with given options
| 
| | mb->load_file(meshFile.c_str(), &rootset, read_options.c_str());
| 

| | // Get all entities of dimension = dim
| 
| | Range elemRange, owned_entities;
| 
| | int dim = 2;
| 
| | mb->get_entities_by_dimension(rootset, dim, elemRange, false);
| 

| | pcomm->filter_pstatus(elemRange, // pass entities that we want to
| | filter
| 
| | PSTATUS_NOT_OWNED, // status we are looking for
| 
| | PSTATUS_NOT, // operation applied ; so it will return owned
| | entities
| | (!not_owned = owned)
| 
| | -1, // this means all processors
| 
| | &owned_entities);
| 

| | std::vector<int> procID(owned_entities.size(), myRank);
| 

| | std::cout << "owned_entities[" << myRank << "], size = " <<
| | owned_entities.size() << std::endl;
| 

| | Tag procID_tag;
| 
| | mb->tag_get_handle("PROC_ID", 1, MB_TYPE_INTEGER, procID_tag,
| | MB_TAG_CREAT | MB_TAG_DENSE, &procID[0]);
| 

| | mb->tag_set_data(procID_tag, owned_entities, &procID[0]);
| 

| | // WRITE_PART writes all partitions to a single output file (only
| | h5m
| | format supports parallel IO at the moment).
| 
| | // One can use the mbconvert tool to convert the output to other
| | formats.
| 
| | mb->write_file(outputFile.c_str(), "H5M", "PARALLEL=WRITE_PART");
| 

| | On Thu, Oct 10, 2013 at 10:37 AM, Tim Tautges < tautges at mcs.anl.gov
| | >
| | wrote:
| 

| | | Yeah, too complicated to backport, and latest works for Carlos
| | | anyway.
| | 
| 

| | | - tim
| | 
| 

| | | On 10/09/2013 09:41 PM, Iulian Grindeanu wrote:
| | 
| 

| | | | ------------------------------ ------------------------------
| | | | ------------------------------ ------------------------------
| | | 
| | 
| 

| | | | Both worked for me, with the current code (4.7.0pre)
| | | 
| | 
| 

| | | | I don't get your error :(
| | | 
| | 
| 
| | | | What version are you using? I will try 4.6.2, but it should be
| | | | fine
| | | | there too :(
| | | 
| | 
| 

| | | | OK, I got an error with more quads, on 4.6.2 maybe I mixed them
| | | | up
| | | | when I saved :(
| | | 
| | 
| 
| | | | mpiexec -np 2 /home/iulian/source/MOAB46/ tools/mbconvert -O
| | | | PARALLEL=READ_PART -O PARTITION=PARALLEL_PARTITION -O
| | | 
| | 
| 
| | | | PARALLEL_RESOLVE_SHARED_ENTS -O PARALLEL_GHOSTS=2.0.1 -o
| | | | PARALLEL=WRITE_PART
| | | 
| | 
| 
| | | | /home/iulian/tmp/2d_naca0012_ mixed2.h5m 2.h5m
| | | 
| | 
| 
| | | | Leaked HDF5 object handle in function at
| | | | ../../../moab46source/src/io/ ReadHDF5.cpp:1523
| | | 
| | 
| 
| | | | Open at entrance: 1
| | | 
| | 
| 
| | | | Open at exit: 2
| | | 
| | 
| 
| | | | Leaked HDF5 object handle in function at
| | | | ../../../moab46source/src/io/ ReadHDF5.cpp:827
| | | 
| | 
| 
| | | | Open at entrance: 1
| | | 
| | 
| 
| | | | Open at exit: 2
| | | 
| | 
| 
| | | | Failed to load "/home/iulian/tmp/2d_naca0012_ mixed2.h5m".
| | | 
| | 
| 
| | | | Error code: MB_INDEX_OUT_OF_RANGE (1)
| | | 
| | 
| 
| | | | Error message: Failed in step PARALLEL READ PART
| | | 
| | 
| 
| | | | Cannot close file with open handles: 0 file, 1 data, 0 group, 0
| | | | type,
| | | | 0 attr
| | | 
| | 
| 

| | | | I will look into it.
| | | 
| | 
| 

| | | | Hi Carlos,
| | | 
| | 
| 
| | | | It looks like it is a bug in 4.6.2.
| | | 
| | 
| 
| | | | I don't know if it will be fixed, there are some important
| | | | changes
| | | | in
| | | | ghosting in current version.
| | | 
| | 
| 
| | | | So for Version4.6 branch, the model with 17 quads works fine if
| | | | you
| | | | don't do ghosting:
| | | 
| | 
| 

| | | | iulian at T520-iuli:~/source/ MOAB46$ mpiexec -np 2
| | | | /home/iulian/source/MOAB46/ tools/mbconvert -O
| | | | PARALLEL=READ_PART
| | | | -O
| | | 
| | 
| 
| | | | PARTITION=PARALLEL_PARTITION -O PARALLEL_RESOLVE_SHARED_ENTS -o
| | | | PARALLEL=WRITE_PART
| | | 
| | 
| 
| | | | /home/iulian/tmp/2d_naca0012_ mixed_invert.h5m 2.h5m
| | | 
| | 
| 
| | | | Read "/home/iulian/tmp/2d_naca0012_ mixed_invert.h5m"
| | | 
| | 
| 
| | | | Wrote "2.h5m"
| | | 
| | 
| 

| | | | I would recommend upgrading to current version.
| | | 
| | 
| 
| | | | That code is pretty complicated, and I am not sure if we will
| | | | backport changes to Version4.6 branch.
| | | 
| | 
| 

| | | | Tim, what do you suggest? Should I try to backport some changes
| | | | in
| | | | ParallelComm? I know you are working on that code.
| | | 
| | 
| 

| | | | Thanks,
| | | 
| | 
| 
| | | | Iulian
| | | 
| | 
| 

| | | --
| | 
| 
| | | ============================== ==============================
| | | ====
| | 
| 
| | | "You will keep in perfect peace him whose mind is
| | 
| 
| | | steadfast, because he trusts in you." Isaiah 26:3
| | 
| 

| | | Tim Tautges Argonne National Laboratory
| | 
| 
| | | ( tautges at mcs.anl.gov ) (telecommuting from UW-Madison)
| | 
| 
| | | phone (gvoice): (608) 354-1459 1500 Engineering Dr.
| | 
| 
| | | fax: (608) 263-4499 Madison, WI 53706
| | 
| 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/moab-dev/attachments/20131012/06537577/attachment.html>