<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Times New Roman; font-size: 12pt; color: #000000'>Hello,<br>Thank you for the detailed report.<br>Sorry for the silence, my email didn't work.<br>I am using parmetis 3.1 in my build, it may take a while to reproduce your error. I will try compiling your code and let you know.<br><br>Still, what kind of capability you need? Is mbpart not enough for you? Or do you want to do partitioning in the same run?<br><br>Thanks,<br>Iulian<br><br><br><hr id="zwchr"><blockquote style="border-left:2px solid rgb(16, 16, 255);margin-left:5px;padding-left:5px;"><div id="DWT199"><br></div></blockquote><br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px;"><div></div><div>By clearing ranges</div><div>ParallelComm::SharedEnts</div><div>ParallelComm::interfaceSets</div><div>and calling function ParallelComm::reset_all_buffers</div><div>I've passed resolve_shared_ents function and now </div><div>have an error in exchange_ghost_cells:</div><div><br></div><div><div>Invalid entity handle: Hex 550</div><div>Couldn't get pstatus tag.</div><div> cant get_sharing_data </div><div>Trouble setting remote data range on sent entities in ghost exchange.</div><div>Failed to unpack remote handles.</div></div><div><br></div><div>or</div><div><br></div><div>Found bad entities in check_local_shared, proc rank 5, </div><div>...</div><div>Reason: Vertex proc set not same size as entity proc set.</div><div><br></div><div><br></div><div><br></div><div>Probably there are some other stored ranges that should be cleared?</div><br><div><div>On Apr 14, 2012, at 6:35 AM, Iulian Grindeanu wrote:</div><br class="Apple-interchange-newline"><blockquote><div><div style="font-family: 'Times New Roman'; font-size: 12pt; color: rgb(0, 0, 0);">Hello<br><br><hr id="zwchr"><blockquote id="DWT237" style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px;">Hello,<br>I'm trying to implement distribution of entities between multiple processors based on parmetis graph partitioning in parallel.<br>I've used moab #4.5.0 and #4.6.0pre<br><br></blockquote>Can you try current dev version?<br></div></div></blockquote><div><br></div><div>I've downloaded moab from <a href="http://ftp.mcs.anl.gov/pub/fathom/moab-nightly.tar.gz" target="_blank">http://ftp.mcs.anl.gov/pub/fathom/moab-nightly.tar.gz</a> which appeared to be 4.6.0pre. </div><div id="DWT187">Is it current dev version?</div></div></blockquote>this is the current dev version<br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px;"><div><div></div><br><blockquote><div><div style="font-family: 'Times New Roman'; font-size: 12pt; color: rgb(0, 0, 0);"><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; position: static; z-index: auto;">I've tested different approaches for exchanging entities which are<br>1. exchange_owned_meshs - hangs in cycle while(incoming2)<br>2. send_entities / recv_entities - works if # of processors is 2 otherwise hangs<br>3. pack_buffer / unpack_buffer<br><br>3rd one work if i don't set "store_remote_handles" flag, because otherwise it stops on following assertion while unpacking new entities:<br>Assertion failed: (new_ents[ind] == from_vec[i]), function get_remote_handles, file ParallelComm.cpp, line 1728.<br><br></blockquote>Can you send test files for us to be able to reproduce the error?<br></div></div></blockquote><div><br></div><div><br></div>I attach source file test.cpp which may be compiled by</div><div>mpicxx test.cpp -lmoab -lnetcdf -lparmetis -lmetis</div><div>and run for example by</div><div>mpirun -n 10 mesh1.mhdf 0</div><div>last number may be 0,1,2</div><div>0 - load file on one processor</div><div>1 - load already distributed file (for example by mbpart)</div><div>2 - load file on every processor then determine own part of </div><div>the mesh by processor rank and elements global id.</div><div><br></div><div>It is written for parmetis 4.0.2</div><div>it may be compiled with</div><div>-DUSE_EXCHANGE_OWNED_MESHES - to use </div><div>exchange_owned_meshs for entities distribution</div><div>-DUSE_SENDRECV - to use send_entities / recv_entities</div><div>-DSTORE_REMOTE_HANDLES - set store_remote_handles </div><div>flag in pack_buffer / unpack_buffer and exchange </div><div>remote handles between processors</div><div>-DDEBUG_OUTPUT - to let program output information about distribution</div><div><br></div><div>I attach two files with mesh:</div><div>mesh1.mhdf - serial file with mesh</div><div>mesh.mhdf - same mesh, but distributed for 4 processors by mbpart</div><div><br><blockquote><div><div style="font-family: 'Times New Roman'; font-size: 12pt; color: rgb(0, 0, 0);"><blockquote id="DWT238" style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; position: static; z-index: auto;">so if i don't use store_remote_handles I obtain final distribution.<br>it works well if load mesh on one processor and distribute<span class="Apple-converted-space"> </span><br>it across many processors.<span class="Apple-converted-space"> </span><br>But if i load already distributed mesh, program fails on resolve_shared_ents with following errors:<br><br>Invalid entity handle: Vertex 21<br>Failed to get pstatus flag.<br>Couldn't filter on owner.<br>get_sent_ents failed.<br>Trouble resolving shared entity remote handles.<br>cannot resolve ents<br><br></blockquote>how do you load? what options do you use? How is the mesh partitioned?<br></div></div></blockquote><div><br></div><div>When I load mesh already distributed by mbpart I use</div><div>"PARALLEL=READ_PART;</div><div>PARTITION=PARALLEL_PARTITION;</div><div>PARTITION_BY_RANK;</div><div>PARALLEL_COMM=xxx;", where xxx is id of communicator</div><div><br></div><div>Otherwise I load file without any options on one processor or on </div><div>all processors then in second case I erase part of the mesh </div><div>and then in both cases create a set with PARALLEL_PARTITION tag</div><br><blockquote><div><div style="font-family: 'Times New Roman'; font-size: 12pt; color: rgb(0, 0, 0);"><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; position: static; z-index: auto;">Which is strange, because after distribution i set for all entities of all dimensions zeros for pstatus_tag, zeros for sharedh_tag and -1 for sharedp_tag<br><br>Could you help me with that problem?<br><br><br><br><br></blockquote><br></div></div><br class="Apple-interchange-newline"></blockquote></div><br><br><br></blockquote><br></div></body></html>