<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000'>Thank you Carlos and Junior, for your help.<br>I will need to build cgns library; you mentioned that you tested 3.2 beta, too; I will probably stick with 3.1.4, (release 2?). <br> Also, what version of hdf5 library do you use? Should it work for any version greater than 1.8? <br>I assume you build statically? Can I also get your config.log files for cgns and moab?<br><br>I assume that you are not using any of the parallel io capability, because that seems to be available only in beta version. <br>So everything is read/written in serial, isn't it?<br>I am not sure if we will port the cgns reader / writer to 4.6.2 also. Is your patch against 4.6.2 tar file or against "Version4.6" branch? There should not be much difference. <br><br>Tim, what do you suggest? In a way, it should be easier to update 4.6.2. But I don't want to merge to the master after that, there are too many changes. This patch should work out of the box for 4.6.2, after I build cgns.<br><br>Best Regards,<br>Iulian<br><br><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div><div>Iulian,<br><br></div>Sorry for the delay in getting back to you... I have tested the new version and it worked OK for my mesh sample. I will test it latter for other cases.<br><br>Also, I am sending you the CGNS reader and writer along with input cgns mesh examples. We have worked on top of version 462, so things may look a little out of place from the repository.<br>
<br></div><div>The attached patch modifies some autotools files and new files (apply with patch -p0 -i moab462_cgns.patch). Moreover, the tar.gz contains sample meshes (simple airfoil mesh in 2D and a mixed-mesh discretization of a 3D wingbody geometry).<br>
<br></div><div>We have tested the CGNS capability with the latest library vesion 3.1.4 and 3.2-beta. Furthermore, we sticked with the base CGNS use-case, that is, single base/zone. We will also implement latter capability to write out user variables.<br>
<br></div><div>Please, feel free to improve the files and if you have any questions, let us know. <a href="mailto:carbrevi@gmail.com" target="_blank">carbrevi@gmail.com</a> and <a href="mailto:junior.hmg@gmail.com" target="_blank">junior.hmg@gmail.com</a><br>
<br>Regards,<br><br>Carlos<br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Nov 16, 2013 at 3:57 AM, Iulian Grindeanu <span dir="ltr"><<a href="mailto:iulian@mcs.anl.gov" target="_blank">iulian@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:times new roman,new york,times,serif">Hi Carlos,<br>That ticket is now closed, Tim fixed the bug.<br>
Please let us know if it works for you.<br><br>Thanks for your patience,<br>Iulian<br><hr><div><div class="h5"><blockquote style="padding-left:5px;font-size:12pt;font-style:normal;margin-left:5px;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal;border-left:2px solid #1010ff">
<div dir="ltr"><div><div>Iulian,<br><br></div>thanks for checking this out. I will be following the trac ticket. The mesh I used is attached, partitioned for 8 procs (all tri). I noticed that are much simpler sample meshes you created to investigate this issue. Anyway, you can use the airfoil mesh if you find it useful.<br>
<br>I use a separate code to compute elements adjacency and then pass those to metis. To write the partitions into h5m I followed the code form mbzoltan tool. The partitioning code is correct (element-based) since I have used this code previously with other applications...<br>
<br></div>I am from Brazil indeed, utc -3h. I'm a PhD candidate from Technological Institute of Aeronautics in Sao Jose dos Campos.<br><div><div><br></div><div>Now I am building/adapting parts of my research code (high-order unstructured CFD) to use MOAB. I will stick with 462 for this, working with non-mixed meshes for this transition period. In the near feature I will also look into the structured meshes capability of the library, as well as the high-order elements implementation.<br>
<br></div><div>The CGNS reader/writer is in the works too and I will submit those as well. My co-worker Junior, cc'ed here, is dealing with the writer part and we are almos done.<br></div><div><br></div><div>Regards,<br>
Carlos Breviglieri<br></div><div><br><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Oct 12, 2013 at 5:40 PM, Iulian Grindeanu <span dir="ltr"><<a href="mailto:iulian@mcs.anl.gov" target="_blank">iulian@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:times new roman,new york,times,serif">Hi Carlos,<br>I am seeing a problem with ghosting, thank you for pointing it out. Danqing and I messed up with that code :( to fix some other issues we were seeing. <br>
<a href="http://trac.mcs.anl.gov/projects/ITAPS/ticket/284" target="_blank">http://trac.mcs.anl.gov/projects/ITAPS/ticket/284</a><br><br>It may take a while to fix it properly.<br>Thanks again,<br>Iulian<br><br><hr><div>
<div>
<blockquote style="padding-left:5px;font-size:12pt;font-style:normal;margin-left:5px;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal;border-left:2px solid #1010ff"><div style="font-size:12pt;font-family:times new roman,new york,times,serif">
Hmmmm,<br>Can you send me the 2d_naca0012.h5m with your partition? <br>If you do ghosting after reading, did you get the same results?<br>The elements/processors should not change ownership after ghosting, but maybe they do. <br>
It is indeed a work in progress, you may have found another issue :(<br>Not being able to write is probably the biggest problem.<br> <br>On what time zone do you work? :) Also, where are you from? The name looks Brazilian, Portuguese, or Italian ? Or Spanish? <br>
<br>Thanks,<br>Iulian<br><br><hr><blockquote style="padding-left:5px;font-size:12pt;font-style:normal;margin-left:5px;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal;border-left:2px solid #1010ff">
<div dir="ltr"><div><div><div><div><div><div>Hi Iulian,<br><br></div>I have just ran some tests with the latest clone of the master repo (saturday morning). Here are my findings with MOAB 470pre:<br><br></div>Now I am able to read partitined mixed meshes without errors. However, if I plot the owned entitied for a given proc, even for non-mixed meshes, they are different from the ones obtained by the partitioner, see below<br>
<br></div>Mesh distribution for 2d_naca0012.h5m (all tri mesh) from partitioner, over 8 procs:<br>proc[0] has 866 elements<br>proc[1] has 863 elements<br>proc[2] has 866 elements<br>proc[3] has 869 elements<br>proc[4] has 872 elements<br>
proc[5] has 869 elements<br>proc[6] has 862 elements<br>proc[7] has 877 elements<br><br></div>Mesh distribution seen by MOAB 470pre ("PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS;PARALLEL_GHOSTS=2.0.1")<br>
owned_entities[5], size = 864<br>owned_entities[6], size = 853<br>owned_entities[7], size = 871<br>owned_entities[1], size = 859<br>owned_entities[3], size = 860<br>owned_entities[4], size = 867<br>owned_entities[0], size = 866<br>
owned_entities[2], size = 861<br><br></div>besides procs 0, all other report wrong number of entities. Code to compute such distribution is below (based on example/HelloParMOAB.cpp). The mixed mesh (2d_naca0012_mixed.h5m) now is read (does not crash) but the distribution is off as well. With MOAB 462 the distribution is OK for homogeneous meshes.<br>
<br></div>Moreover, with MOAB 470pre, no output is written to disk with PARALLEL=WRITE_PART, regardless of the mesh type. Using PARALLEL=NONE works, but only one part of the domain is written, as expected.<br><div><br></div>
<div>I understand that this is a work in progress. If you need more information, let me know.<br></div><div><br>Regards,<br><br>Carlos Breviglieri<br><br><br> read_options = "PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS;PARALLEL_GHOSTS=2.0.1";<br>
<br> moab::Interface* mb = new Core;<br><br> // Create root sets for each mesh. Then pass these<br> // to the load_file functions to be populated.<br> EntityHandle rootset, partnset;<br> mb->create_meshset(MESHSET_SET, rootset);<br>
mb->create_meshset(MESHSET_SET, partnset);<br><br> // Create the parallel communicator object with the partition handle associated with MOAB<br> ParallelComm *pcomm = ParallelComm::get_pcomm(mb, partnset, &myComm);<br>
<br> // Load the file from disk with given options<br> mb->load_file(meshFile.c_str(), &rootset, read_options.c_str());<br><br> // Get all entities of dimension = dim<br> Range elemRange, owned_entities;<br>
int dim = 2;<br> mb->get_entities_by_dimension(rootset, dim, elemRange, false);<br><br> pcomm->filter_pstatus(elemRange, // pass entities that we want to filter<br> PSTATUS_NOT_OWNED, // status we are looking for<br>
PSTATUS_NOT, // operation applied ; so it will return owned entities (!not_owned = owned)<br> -1, // this means all processors<br> &owned_entities);<br>
<br> std::vector<int> procID(owned_entities.size(), myRank);<br><br> std::cout << "owned_entities[" << myRank << "], size = " << owned_entities.size() << std::endl;<br>
<br> Tag procID_tag;<br> mb->tag_get_handle("PROC_ID", 1, MB_TYPE_INTEGER, procID_tag, MB_TAG_CREAT | MB_TAG_DENSE, &procID[0]);<br><br> mb->tag_set_data(procID_tag, owned_entities, &procID[0]);<br>
<br> // WRITE_PART writes all partitions to a single output file (only h5m format supports parallel IO at the moment).<br> // One can use the mbconvert tool to convert the output to other formats.<br> mb->write_file(outputFile.c_str(), "H5M", "PARALLEL=WRITE_PART");<br>
<br><br><div><br><br><br><div><div><br><div><div><div><br><br></div></div></div></div></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Oct 10, 2013 at 10:37 AM, Tim Tautges <span dir="ltr"><<a href="mailto:tautges@mcs.anl.gov" target="_blank">tautges@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Yeah, too complicated to backport, and latest works for Carlos anyway.<br>
<br>
- tim<br>
<br>
On 10/09/2013 09:41 PM, Iulian Grindeanu wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
------------------------------<u></u>------------------------------<u></u>------------------------------<u></u>------------------------------<div><div><br>
<br>
Both worked for me, with the current code (4.7.0pre)<br>
<br>
I don't get your error :(<br>
What version are you using? I will try 4.6.2, but it should be fine there too :(<br>
<br>
<br>
OK, I got an error with more quads, on 4.6.2 maybe I mixed them up when I saved :(<br>
mpiexec -np 2 /home/iulian/source/MOAB46/<u></u>tools/mbconvert -O PARALLEL=READ_PART -O PARTITION=PARALLEL_PARTITION -O<br>
PARALLEL_RESOLVE_SHARED_ENTS -O PARALLEL_GHOSTS=2.0.1 -o PARALLEL=WRITE_PART<br>
/home/iulian/tmp/2d_naca0012_<u></u>mixed2.h5m 2.h5m<br>
Leaked HDF5 object handle in function at ../../../moab46source/src/io/<u></u>ReadHDF5.cpp:1523<br>
Open at entrance: 1<br>
Open at exit: 2<br>
Leaked HDF5 object handle in function at ../../../moab46source/src/io/<u></u>ReadHDF5.cpp:827<br>
Open at entrance: 1<br>
Open at exit: 2<br>
Failed to load "/home/iulian/tmp/2d_naca0012_<u></u>mixed2.h5m".<br>
Error code: MB_INDEX_OUT_OF_RANGE (1)<br>
Error message: Failed in step PARALLEL READ PART<br>
Cannot close file with open handles: 0 file, 1 data, 0 group, 0 type, 0 attr<br>
<br>
<br>
I will look into it.<br>
<br>
Hi Carlos,<br>
It looks like it is a bug in 4.6.2.<br>
I don't know if it will be fixed, there are some important changes in ghosting in current version.<br>
So for Version4.6 branch, the model with 17 quads works fine if you don't do ghosting:<br>
<br>
iulian@T520-iuli:~/source/<u></u>MOAB46$ mpiexec -np 2 /home/iulian/source/MOAB46/<u></u>tools/mbconvert -O PARALLEL=READ_PART -O<br>
PARTITION=PARALLEL_PARTITION -O PARALLEL_RESOLVE_SHARED_ENTS -o PARALLEL=WRITE_PART<br>
/home/iulian/tmp/2d_naca0012_<u></u>mixed_invert.h5m 2.h5m<br>
Read "/home/iulian/tmp/2d_naca0012_<u></u>mixed_invert.h5m"<br>
Wrote "2.h5m"<br>
<br>
I would recommend upgrading to current version.<br>
That code is pretty complicated, and I am not sure if we will backport changes to Version4.6 branch.<br>
<br>
Tim, what do you suggest? Should I try to backport some changes in ParallelComm? I know you are working on that code.<br>
<br>
Thanks,<br>
Iulian<br>
<br>
<br>
<br>
<br>
<br>
</div></div></blockquote><div><div>
<br>
-- <br>
==============================<u></u>==============================<u></u>====<br>
"You will keep in perfect peace him whose mind is<br>
steadfast, because he trusts in you." Isaiah 26:3<br>
<br>
Tim Tautges Argonne National Laboratory<br>
(<a href="mailto:tautges@mcs.anl.gov" target="_blank">tautges@mcs.anl.gov</a>) (telecommuting from UW-Madison)<br>
phone (gvoice): <a href="tel:%28608%29%20354-1459" target="_blank">(608) 354-1459</a> 1500 Engineering Dr.<br>
fax: <a href="tel:%28608%29%20263-4499" target="_blank">(608) 263-4499</a> Madison, WI 53706<br>
<br>
</div></div></blockquote></div><br></div>
</blockquote><br></div></blockquote><br></div></div></div></div></blockquote></div><br></div>
</blockquote><br></div></div></div></div></blockquote></div><br></div>
</blockquote><br></div></body></html>