On Wed, Apr 11, 2012 at 10:49 PM, Mohammad Mirzadeh <span dir="ltr"><<a href="mailto:mirzadeh@gmail.com">mirzadeh@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I see. Well the reason for non-symmetry is that I'm using a finite difference discretization on quadtree AMR grids. When discretizing at a hanging node, i.e. a node that is missing a direct neighbor, I use interpolation from nodes of the parent cell. When it comes to discretizing at those nodes (i.e. nodes of the parent), however, they are on a coarser grid and will have direct neighbors so they would not see the hanging node in their discretization. This leads to a non-symmetric discretization matrix which I was using (structurally) to define the adjacency matrix of the graph. I think I can fix this by requiring the parent nodes to also have a link to the hanging node (although they are not used in the discretization anyway)<div>
<br></div><div>I'm just surprised that ParMetis does not complain about non-symmetric graphs at lower resolutions and partitions them just fine!</div></div></blockquote><div><br></div><div>ParMetis does not burden the user with a whole lot of diagnostic output :)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>As for the debugger, -start_in_debugger does not start the debugger for me :(. I'm using mpirun -np 2 ./a.out -start_in_debugger. Is this the correct way?</div>
</div></blockquote><div><br></div><div>Send the entire output. It is likely that at the beginning there are lines showing a failure to open an X window.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><br><div class="gmail_quote">On Wed, Apr 11, 2012 at 3:07 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On Wed, Apr 11, 2012 at 8:24 PM, Mohammad Mirzadeh <span dir="ltr"><<a href="mailto:mirzadeh@gmail.com" target="_blank">mirzadeh@gmail.com</a>></span> wrote:<br></div><div class="gmail_quote"><div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I was reading the FAQ list when I came across the following:<div><br></div><div><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#key" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#key</a></div>
<div><br></div><div><h3 style="font-family:'Times New Roman';background-color:rgb(213,234,255)"><a name="136a3999c18fa0b7_136a3724a05f6aec_136a314e6dfe28a7_key" style="color:red">When calling MatPartitioningApply() you get a message Error! Key 16615 not found</a></h3>
<span style="font-family:'Times New Roman';background-color:rgb(213,234,255);font-size:medium">The graph of the matrix you are using is not symmetric. You must use symmetric matrices for partitioning.</span></div>
<div><font face="'Times New Roman'" size="3"><br></font></div><div><font face="'Times New Roman'" size="3">Is this a limitation on ParMetis side? I set up the adjacency matrix based on the discretization that I will be performing on the grid which is non-symmetric; both numerically and structurally. What's the solution here? Make an "approximate" adjacency matrix that sort of looks like (structurally) my discretization but is symmetric? What I don't understand is my matrix IS non-symmetric when the code runs on coarser grids!</font></div>
</div></blockquote><div><br></div></div><div>I don't quite understand how you can have a non-symmetric adjacency description. But Metis/ParMetis partitions undirected graphs, which</div><div>by definition have symmetric adjacency matrices. Non-symmetric adjacency would seem to imply a directed graph of some sort, or more plainly,</div>
<div>something is a adjacent to another thing which is not adjacent to it. That is a very strange concept.</div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><font face="'Times New Roman'" size="3">Also, I was reading the FAQ hoping I can find something regarding using gdb in parallel. I found this: <a href="http://scicomp.stackexchange.com/a/410/485" target="_blank">http://scicomp.stackexchange.com/a/410/485</a> but I'm not sure how I should be using gdb in parallel. Could you (maybe Matt?) please explain a little bit?</font></div>
</div></blockquote><div><br></div></div><div>-start_in_debugger spawns a gdb windows for EVERY process and attaches it</div><div><br></div><div>-debbuger_nodes a,b,c spawns gdb windows ONLY for ranks a, b, and c</div><div>
<br></div>
<div> Thanks,</div><div><br></div><div> Matt</div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><font face="'Times New Roman'" size="3">Thanks<br>
</font><br><div class="gmail_quote">On Tue, Apr 10, 2012 at 12:08 AM, Mohammad Mirzadeh <span dir="ltr"><<a href="mailto:mirzadeh@gmail.com" target="_blank">mirzadeh@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Just built petsc-dev and it did not help. I'm going to look into the code to see if my graph is "ill-formed" in some sense. Just hope the problem is from my side not a real bug in ParMetis!<div>
<div><div>
<br></div><div> </div><div><br><div class="gmail_quote">On Mon, Apr 9, 2012 at 8:46 PM, Mohammad Mirzadeh <span dir="ltr"><<a href="mailto:mirzadeh@gmail.com" target="_blank">mirzadeh@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">ok. Thanks Jed. I'll try petsc-dev to see if it fixes the problem.<div><br></div><div>Thanks everyone.<div><div><br><br><div class="gmail_quote">On Mon, Apr 9, 2012 at 8:42 PM, Jed Brown <span dir="ltr"><<a href="mailto:jedbrown@mcs.anl.gov" target="_blank">jedbrown@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_quote"><div>On Mon, Apr 9, 2012 at 22:37, Mohammad Mirzadeh <span dir="ltr"><<a href="mailto:mirzadeh@gmail.com" target="_blank">mirzadeh@gmail.com</a>></span> wrote:<br>
</div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Thanks Sean. I'm using Petsc 3.2-p6 along with ParMetis 4.0.2. Since this was not supported with 3.2-p6, and previous versions had bugs, I built parmetis myself and used --with-parmetis-include and --with-parmetis-lib flags to build petsc.<div>
<br></div><div>Should I switch to petsc-dev?</div></div></blockquote></div></div><br><div>Yes, and use --download-metis --download-parmetis because the version upstream has some bugs for which the patches have not been applied.</div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div></div>
</blockquote></div></div><span><font color="#888888"><br><br clear="all"><span class="HOEnZb"><font color="#888888"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</font></span></font></span></blockquote></div><br></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>