<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000'>Hi Carlos,<br>I am seeing a problem with ghosting, thank you for pointing it out. Danqing and I messed up with that code :( to fix some other issues we were seeing. <br>http://trac.mcs.anl.gov/projects/ITAPS/ticket/284<br><br>It may take a while to fix it properly.<br>Thanks again,<br>Iulian<br><br><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><style>p { margin: 0; }</style><div style="font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000">Hmmmm,<br>Can you send me the 2d_naca0012.h5m with your partition? <br>If you do ghosting after reading, did you get the same results?<br>The elements/processors should not change ownership after ghosting, but maybe they do. <br>It is indeed a work in progress, you may have found another issue :(<br>Not being able to write is probably the biggest problem.<br> <br>On what time zone do you work? :) Also, where are you from? The name looks Brazilian, Portuguese, or Italian ? Or Spanish? <br><br>Thanks,<br>Iulian<br><br><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div><div><div><div><div><div>Hi Iulian,<br><br></div>I have just ran some tests with the latest clone of the master repo (saturday morning). Here are my findings with MOAB 470pre:<br><br></div>Now I am able to read partitined mixed meshes without errors. However, if I plot the owned entitied for a given proc, even for non-mixed meshes, they are different from the ones obtained by the partitioner, see below<br>
<br></div>Mesh distribution for 2d_naca0012.h5m (all tri mesh) from partitioner, over 8 procs:<br>proc[0] has 866 elements<br>proc[1] has 863 elements<br>proc[2] has 866 elements<br>proc[3] has 869 elements<br>proc[4] has 872 elements<br>
proc[5] has 869 elements<br>proc[6] has 862 elements<br>proc[7] has 877 elements<br><br></div>Mesh distribution seen by MOAB 470pre ("PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS;PARALLEL_GHOSTS=2.0.1")<br>
owned_entities[5], size = 864<br>owned_entities[6], size = 853<br>owned_entities[7], size = 871<br>owned_entities[1], size = 859<br>owned_entities[3], size = 860<br>owned_entities[4], size = 867<br>owned_entities[0], size = 866<br>
owned_entities[2], size = 861<br><br></div>besides procs 0, all other report wrong number of entities. Code to compute such distribution is below (based on example/HelloParMOAB.cpp). The mixed mesh (2d_naca0012_mixed.h5m) now is read (does not crash) but the distribution is off as well. With MOAB 462 the distribution is OK for homogeneous meshes.<br>
<br></div>Moreover, with MOAB 470pre, no output is written to disk with PARALLEL=WRITE_PART, regardless of the mesh type. Using PARALLEL=NONE works, but only one part of the domain is written, as expected.<br><div><br></div>
<div>I understand that this is a work in progress. If you need more information, let me know.<br></div><div><br>Regards,<br><br>Carlos Breviglieri<br><br><br> read_options = "PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS;PARALLEL_GHOSTS=2.0.1";<br>
<br> moab::Interface* mb = new Core;<br><br> // Create root sets for each mesh. Then pass these<br> // to the load_file functions to be populated.<br> EntityHandle rootset, partnset;<br> mb->create_meshset(MESHSET_SET, rootset);<br>
mb->create_meshset(MESHSET_SET, partnset);<br><br> // Create the parallel communicator object with the partition handle associated with MOAB<br> ParallelComm *pcomm = ParallelComm::get_pcomm(mb, partnset, &myComm);<br>
<br> // Load the file from disk with given options<br> mb->load_file(meshFile.c_str(), &rootset, read_options.c_str());<br><br> // Get all entities of dimension = dim<br> Range elemRange, owned_entities;<br>
int dim = 2;<br> mb->get_entities_by_dimension(rootset, dim, elemRange, false);<br><br> pcomm->filter_pstatus(elemRange, // pass entities that we want to filter<br> PSTATUS_NOT_OWNED, // status we are looking for<br>
PSTATUS_NOT, // operation applied ; so it will return owned entities (!not_owned = owned)<br> -1, // this means all processors<br> &owned_entities);<br>
<br> std::vector<int> procID(owned_entities.size(), myRank);<br><br> std::cout << "owned_entities[" << myRank << "], size = " << owned_entities.size() << std::endl;<br>
<br> Tag procID_tag;<br> mb->tag_get_handle("PROC_ID", 1, MB_TYPE_INTEGER, procID_tag, MB_TAG_CREAT | MB_TAG_DENSE, &procID[0]);<br><br> mb->tag_set_data(procID_tag, owned_entities, &procID[0]);<br>
<br> // WRITE_PART writes all partitions to a single output file (only h5m format supports parallel IO at the moment).<br> // One can use the mbconvert tool to convert the output to other formats.<br> mb->write_file(outputFile.c_str(), "H5M", "PARALLEL=WRITE_PART");<br>
<br><br><div><br><br><br><div><div><br><div><div><div><br><br></div></div></div></div></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Oct 10, 2013 at 10:37 AM, Tim Tautges <span dir="ltr"><<a href="mailto:tautges@mcs.anl.gov" target="_blank">tautges@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Yeah, too complicated to backport, and latest works for Carlos anyway.<br>
<br>
- tim<br>
<br>
On 10/09/2013 09:41 PM, Iulian Grindeanu wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
------------------------------<u></u>------------------------------<u></u>------------------------------<u></u>------------------------------<div><div class="h5"><br>
<br>
Both worked for me, with the current code (4.7.0pre)<br>
<br>
I don't get your error :(<br>
What version are you using? I will try 4.6.2, but it should be fine there too :(<br>
<br>
<br>
OK, I got an error with more quads, on 4.6.2 maybe I mixed them up when I saved :(<br>
mpiexec -np 2 /home/iulian/source/MOAB46/<u></u>tools/mbconvert -O PARALLEL=READ_PART -O PARTITION=PARALLEL_PARTITION -O<br>
PARALLEL_RESOLVE_SHARED_ENTS -O PARALLEL_GHOSTS=2.0.1 -o PARALLEL=WRITE_PART<br>
/home/iulian/tmp/2d_naca0012_<u></u>mixed2.h5m 2.h5m<br>
Leaked HDF5 object handle in function at ../../../moab46source/src/io/<u></u>ReadHDF5.cpp:1523<br>
Open at entrance: 1<br>
Open at exit: 2<br>
Leaked HDF5 object handle in function at ../../../moab46source/src/io/<u></u>ReadHDF5.cpp:827<br>
Open at entrance: 1<br>
Open at exit: 2<br>
Failed to load "/home/iulian/tmp/2d_naca0012_<u></u>mixed2.h5m".<br>
Error code: MB_INDEX_OUT_OF_RANGE (1)<br>
Error message: Failed in step PARALLEL READ PART<br>
Cannot close file with open handles: 0 file, 1 data, 0 group, 0 type, 0 attr<br>
<br>
<br>
I will look into it.<br>
<br>
Hi Carlos,<br>
It looks like it is a bug in 4.6.2.<br>
I don't know if it will be fixed, there are some important changes in ghosting in current version.<br>
So for Version4.6 branch, the model with 17 quads works fine if you don't do ghosting:<br>
<br>
iulian@T520-iuli:~/source/<u></u>MOAB46$ mpiexec -np 2 /home/iulian/source/MOAB46/<u></u>tools/mbconvert -O PARALLEL=READ_PART -O<br>
PARTITION=PARALLEL_PARTITION -O PARALLEL_RESOLVE_SHARED_ENTS -o PARALLEL=WRITE_PART<br>
/home/iulian/tmp/2d_naca0012_<u></u>mixed_invert.h5m 2.h5m<br>
Read "/home/iulian/tmp/2d_naca0012_<u></u>mixed_invert.h5m"<br>
Wrote "2.h5m"<br>
<br>
I would recommend upgrading to current version.<br>
That code is pretty complicated, and I am not sure if we will backport changes to Version4.6 branch.<br>
<br>
Tim, what do you suggest? Should I try to backport some changes in ParallelComm? I know you are working on that code.<br>
<br>
Thanks,<br>
Iulian<br>
<br>
<br>
<br>
<br>
<br>
</div></div></blockquote><div class="HOEnZb"><div class="h5">
<br>
-- <br>
==============================<u></u>==============================<u></u>====<br>
"You will keep in perfect peace him whose mind is<br>
steadfast, because he trusts in you." Isaiah 26:3<br>
<br>
Tim Tautges Argonne National Laboratory<br>
(<a href="mailto:tautges@mcs.anl.gov" target="_blank">tautges@mcs.anl.gov</a>) (telecommuting from UW-Madison)<br>
phone (gvoice): <a href="tel:%28608%29%20354-1459" target="_blank">(608) 354-1459</a> 1500 Engineering Dr.<br>
fax: <a href="tel:%28608%29%20263-4499" target="_blank">(608) 263-4499</a> Madison, WI 53706<br>
<br>
</div></div></blockquote></div><br></div>
</blockquote><br></div></blockquote><br></div></body></html>