<div dir="ltr">This doesn't have to be specific to MATMPIADJ, and could be orthogonal to changing the MatConvert API (which I still think is a good idea). That is, the linear transformation A is a map from a (distributed) vector space X to another space Y. If X=Y, then A is an endomorphism, and when interpreted as a weighted (directed) graph, the diagonal entries are implicit (and ParMetis can't handle them). If instead, X is different from Y, then the diagonal has no implicit meaning.<div>
<br></div><div style>Should we add a MAT_ENDOMORPHISM option? (Easier name?)</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Jan 26, 2013 at 11:12 AM, Jed Brown <span dir="ltr"><<a href="mailto:jedbrown@mcs.anl.gov" target="_blank">jedbrown@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I am inclined to change the MatConvert() interface to require the caller to have created the dest matrix already (similar to MatLoad). In that case, the user would create a MATMPIADJ and set an option instructing it whether rows and columns denote the same graph vertices (in which case the diagonal is implied) or whether they form disjoint sets in a bipartite graph (in which case the diagonal has no special meaning and thus must be copied faithfully).</div>
<div class="HOEnZb"><div class="h5">
<div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jan 25, 2013 at 6:04 PM, Hammond, Glenn <span dir="ltr"><<a href="mailto:glenn.hammond@pnnl.gov" target="_blank">glenn.hammond@pnnl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Some more detail:<br>
<br>
Bear in mind that the arrays passed to ParMETIS_V3_Mesh2Dual() essentially represent entries within rectangular adjacency matrices:<br>
<br>
N_elements x N vertices<br>
<br>
Or<br>
<br>
N_cells x N connections<br>
<br>
If an element and vertex or cell and connection have the same id, they will have a diagonal entry. When I read in the explicit grid, I set up a AIJ matrix of size N_cells x N_connections. For every connection, I fill in two entries (one for each cell) in the corresponding column. I then convert to an Adj matrix and sent it off to ParMETIS. However, during the MatConvert process, diagonal entries are being removed (which changeset 1bdcc2952450 prevented).<br>
<br>
Thanks,<br>
<br>
Glenn<br>
<div><div><br>
-----Original Message-----<br>
From: Hammond, Glenn<br>
Sent: Friday, January 25, 2013 3:18 PM<br>
To: Barry Smith<br>
Cc: <a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a>; Gautam Bisht (<a href="mailto:gbisht@lbl.gov" target="_blank">gbisht@lbl.gov</a>)<br>
Subject: Diagonal entires in MPIAdj matrices<br>
<br>
Barry,<br>
<br>
I see that the diagonal entries in the MPIAdj matrix that Barry added in changeset 1bdcc2952450 (and requested through the discussion below) were removed at the end of November (Jed's changeset from Nov 30: cl1b2d1f). Since I use the adjacency matrix to decompose and set up connectivity for an explicit grid (a grid where connections are listed explicitly in the input file), I need the diagonal entries back in the adjacency matrix as my Adjacency matrix has num_cells rows and num_connections columns (i.e. a cell id can match a column id and thus an entry on the diagonal).<br>
<br>
Or is there an alternative approach to resolve this issue?<br>
<br>
Thanks,<br>
<br>
Glenn<br>
<br>
-----Original Message-----<br>
From: Barry Smith [mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>]<br>
Sent: Thursday, October 04, 2012 5:33 PM<br>
To: Hammond, Glenn<br>
Cc: <a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a><br>
Subject: Re: [petsc-maint #134803] MatConvert for MATMPIAIJ -> MATMPIADJ<br>
<br>
<br>
Ok, the way to deal with this is to remove the diagonal entries just before calling the "broken" ParMETIS V3 PartMeshKway() and which ever other ones are broken.<br>
<br>
Barry<br>
<br>
On Oct 4, 2012, at 7:02 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
<br>
><br>
> Goddamn that stupid George!<br>
><br>
> Ok, too late, its been pushed so now ParMETIS V3 PartMeshKway will break.<br>
><br>
> We can't have a completely different matrix type just because George is an imbecile who should not be allowed to write code.<br>
><br>
> Barry<br>
><br>
> On Oct 4, 2012, at 4:46 PM, "Hammond, Glenn" <<a href="mailto:glenn.hammond@pnnl.gov" target="_blank">glenn.hammond@pnnl.gov</a>> wrote:<br>
><br>
>> Thanks. I will test this.<br>
>><br>
>> I looked into the ParMETIS docs and ParMETIS_V3_Mesh2Dual() is designed solely for meshes where elements are defined through a list of vertices. In other words, rectangular matrices where diagonal entries will potentially exist.<br>
>><br>
>> However, I fear that altering the MATMPIADJ will lead to bugs elsewhere where diagonals are ignored or should not be considered (e.g. line 65 of src/mat/partition/impls/pmetis/pmetis.c). I fear that this is the issue for ParMETIS V3 PartMeshKway().<br>
>><br>
>> Perhaps a new matrix type, MATMPIMESH where the matrix defines a mesh by elements (rows) and vertices (cols)...? Then, the adjacency matrix does not need to include the diagonal.<br>
>><br>
>> I have to agree that the ParMETIS docs are not clear on this!<br>
>><br>
>> Glenn<br>
>><br>
>><br>
>><br>
>>> -----Original Message-----<br>
>>> From: Barry Smith [mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>]<br>
>>> Sent: Thursday, October 04, 2012 2:12 PM<br>
>>> To: <a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a>; Hammond, Glenn<br>
>>> Subject: Re: [petsc-maint #134803] MatConvert for MATMPIAIJ -><br>
>>> MATMPIADJ<br>
>>><br>
>>><br>
>>> Ok, changed petsc-dev to include diagonals in the adjacency matrix<br>
>>><br>
>>><br>
>>> On Oct 4, 2012, at 1:51 PM, "Hammond, Glenn"<br>
>>> <<a href="mailto:glenn.hammond@pnnl.gov" target="_blank">glenn.hammond@pnnl.gov</a>> wrote:<br>
>>><br>
>>>>> Glenn,<br>
>>>>><br>
>>>>> The reason I wrote it originally to skip diagonals was because<br>
>>>>> Metis/Parmetis crashed if you gave it diagonal entries at that time!<br>
>>>>><br>
>>>>> So, what do we need? Sometimes have diagonal entries, sometime<br>
>>> not?<br>
>>>>> How to know when? Does Metis/parmetis still not like diagonal<br>
>>>>> entries<br>
>>> when<br>
>>>>> partitioning? We can certainly change this but we need more information<br>
>>>>> about use cases and what we should change it to.<br>
>>>><br>
>>>> I propose that we include diagonals in the adjacency matrix. The<br>
>>>> adjacency<br>
>>> matrices that I have been using for several years now with the<br>
>>> implicit mesh formation include the diagonals and ParMetis has not<br>
>>> complained. The question is whether others are using MatConvert and will this affect them.<br>
>>> If we want to be backwards compatible, can we add a flag and a<br>
>>> function call to set the flag?<br>
>>>><br>
>>>> Glenn<br>
>>>><br>
>>>>><br>
>>>>><br>
>>>>> Barry<br>
>>>>><br>
>>>>><br>
>>>>> On Oct 4, 2012, at 12:07 PM, "Hammond, Glenn"<br>
>>>>> <<a href="mailto:glenn.hammond@pnnl.gov" target="_blank">glenn.hammond@pnnl.gov</a>> wrote:<br>
>>>>><br>
>>>>>> PETSc,<br>
>>>>>><br>
>>>>>> I am using MatConvert() to convert a MATMPIAIJ to a MATMPIADJ.<br>
>>>>>> In<br>
>>>>> stepping through the MatConvert() routine (i.e.<br>
>>>>> MatConvertFrom_MPIAdj), I notice that diagonal entries are ignored:<br>
>>>>>><br>
>>>>>><br>
>>>>>> /* count the number of nonzeros per row */ for (i=0; i<m; i++) {<br>
>>>>>> ierr = MatGetRow(A,i+rstart,&len,&rj,PETSC_NULL);CHKERRQ(ierr);<br>
>>>>>> for (j=0; j<len; j++) {<br>
>>>>>> --> if (rj[j] == i+rstart) {len--; break;} /* don't count diagonal */<br>
>>>>>> }<br>
>>>>>> ierr =<br>
>>> MatRestoreRow(A,i+rstart,&len,&rj,PETSC_NULL);CHKERRQ(ierr);<br>
>>>>>> nzeros += len;<br>
>>>>>> }<br>
>>>>>><br>
>>>>>> which would be fine if my matrix were NxN. However, my AIJ<br>
>>>>>> matrix is<br>
>>>>> rectangular (MxN, or # cells X # connections), and therefore, I<br>
>>>>> need diagonal entries in my MATMPIADJ matrix. Am I missing<br>
>>>>> something? I know that several years ago, Barry had to go in and<br>
>>>>> fix MATMPIADJ for rectangular matrices so that they would include the diagonal (see below).<br>
>>>>>><br>
>>>>>> Thanks,<br>
>>>>>><br>
>>>>>> Glenn<br>
>>>>>><br>
>>>>>> From 10/1/2009<br>
>>>>>><br>
>>>>>> Glenn,<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> Ok, I do not understand this properly. I didn't even know this<br>
>>>>>> mesh matrix<br>
>>>>> was rectangular. The Metis users manual is short on detail.<br>
>>>>>><br>
>>>>>> I will have to spend more time to understand what Metis does and<br>
>>>>>> fix up<br>
>>>>> things for you.<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> Barry<br>
>>>>>><br>
>>>>>><br>
>>>><br>
>>>><br>
>><br>
><br>
<br>
<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>