<div dir="ltr">Fande, It looks to me like this branch in ParMetis must be taken to trigger this error. First <b>Match_SHEM</b> and then CreateCoarseGraphNoMask.<div><br><div> /* determine which matching scheme you will use */<br> switch (ctrl->ctype) {<br> case METIS_CTYPE_RM:<br> Match_RM(ctrl, graph);<br> break;<br> case METIS_CTYPE_SHEM:<br> if (eqewgts || graph->nedges == 0)<br> Match_RM(ctrl, graph);<br> else<br><b> Match_SHEM(ctrl, graph);<br></b> break;<br> default:<br> gk_errexit(SIGERR, "Unknown ctype: %d\n", ctrl->ctype);<br> }<br></div><div><br></div><div>-----------------------</div><div><br></div><div> /* Check if the mask-version of the code is a good choice */<br> mask = HTLENGTH;<br> if (cnvtxs < 2*mask || graph->nedges/graph->nvtxs > mask/20) { <br> CreateCoarseGraphNoMask(ctrl, graph, cnvtxs, match);<br> return;<br> }<br></div><div><br></div></div><div>------------</div><div><br></div><div>The actual error is in CreateCoarseGraphNoMask, graph->cmap is too small and this gets garbage. parmetis coarsen.c:856:</div><div><br></div><div> istart = xadj[v];<br> iend = xadj[v+1];<br> for (j=istart; j<iend; j++) {<br><b> k = cmap[adjncy[j]];<br></b> if ((m = htable[k]) == -1) {<br> cadjncy[nedges] = k;<br></div><div><br></div><div> Anyway, this is what I have so far,</div><div>Mark</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Nov 10, 2019 at 10:34 AM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Fande, the problem is k below seems to index beyond the end of htable, resulting in a crazy m and a segv on the last line below.</div><div><br></div><div>I don't have a clean valgrind machine now, that is what is needed if no one has seen anything like this. I could add a test in a MR and get the pipeline to do it.</div><div><br></div>void CreateCoarseGraphNoMask(ctrl_t *ctrl, graph_t *graph, idx_t cnvtxs, <br> idx_t *match)<br>{<br> idx_t j, k, m, istart, iend, nvtxs, nedges, ncon, cnedges, v, u, dovsize;<br> idx_t *xadj, *vwgt, *vsize, *adjncy, *adjwgt;<br> idx_t *cmap, *htable;<br> idx_t *cxadj, *cvwgt, *cvsize, *cadjncy, *cadjwgt;<br> graph_t *cgraph;<br>ine <br> WCOREPUSH;<br><br> dovsize = (ctrl->objtype == METIS_OBJTYPE_VOL ? 1 : 0);<br><br> IFSET(ctrl->dbglvl, METIS_DBG_TIME, gk_startcputimer(ctrl->ContractTmr));<br><br> nvtxs = graph->nvtxs;<br> ncon = graph->ncon;<br> xadj = graph->xadj;<br> vwgt = graph->vwgt;<br> vsize = graph->vsize;<br> adjncy = graph->adjncy;<br> adjwgt = graph->adjwgt;<br> cmap = graph->cmap;<br><br><br> /* Initialize the coarser graph */<br> cgraph = SetupCoarseGraph(graph, cnvtxs, dovsize);<br> cxadj = cgraph->xadj;<br> cvwgt = cgraph->vwgt;<br> cvsize = cgraph->vsize;<br> cadjncy = cgraph->adjncy;<br> cadjwgt = cgraph->adjwgt;<br><br> htable = iset(cnvtxs, -1, iwspacemalloc(ctrl, cnvtxs));<br><br> cxadj[0] = cnvtxs = cnedges = 0;<br> for (v=0; v<nvtxs; v++) {<br> if ((u = match[v]) < v)<br> continue;<br><br> ASSERT(cmap[v] == cnvtxs);<br> ASSERT(cmap[match[v]] == cnvtxs);<br><br> if (ncon == 1)<br> cvwgt[cnvtxs] = vwgt[v];<br> else<br> icopy(ncon, vwgt+v*ncon, cvwgt+cnvtxs*ncon);<br><br> if (dovsize)<br> cvsize[cnvtxs] = vsize[v];<br><br> nedges = 0;<br><br> istart = xadj[v];<br> iend = xadj[v+1];<br> for (j=istart; j<iend; j++) {<br> k = cmap[adjncy[j]];<br> if ((m = htable[k]) == -1) {<br><div> cadjncy[nedges] = k;<br> cadjwgt[nedges] = adjwgt[j];<br> htable[k] = nedges++;<br> }<br> else {<br> cadjwgt[m] += adjwgt[j];<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Nov 10, 2019 at 1:35 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Nov 9, 2019 at 10:51 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank">fdkong.jd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Mark,<div><br></div><div>Thanks for reporting this bug. I was surprised because we have sufficient heavy tests in moose using partition weights and do not have any issue so far.</div><div><br></div></div></blockquote><div><br></div><div>I have been pounding on this code with elasticity and have not seen this issue. I am now looking at Lapacianas and I only see it with pretty large problems. The example below is pretty minimal (eg, it works with 16 cores and it works with -dm_refine 4). I have reproduced this on Cori, SUMMIT and my laptop.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>I will take a shot on this.</div></div></blockquote><div><br></div><div>Thanks, I'll try to take a look at it also. I have seen it in DDT, but did not dig further. It looked like a typical segv in ParMetis.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Fande,</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Nov 9, 2019 at 3:08 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">snes/ex13 is getting a ParMetis segv with GAMG and coarse grid repartitioning. Below shows the branch and how to run it.<div><br></div><div>I've tried valgrind on Cori but it gives a lot of false positives. I've seen this error in DDT but I have not had a chance to dig and try to fix it. At least I know it has something to do with weights.</div><div><br></div><div>If anyone wants to take a shot at it feel free. This bug rarely happens.<br><div></div><div><div><br></div><div>The changes use weights and are just a few lines of code (from 1.5 years ago):</div><div><br></div><div>12:08 (0455fb9fec...)|BISECTING ~/Codes/petsc$ git bisect bad<br>0455fb9fecf69cf5cf35948c84d3837e5a427e2e is the first bad commit<br>commit 0455fb9fecf69cf5cf35948c84d3837e5a427e2e<br>Author: Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank">fdkong.jd@gmail.com</a>><br>Date: Thu Jun 21 18:21:19 2018 -0600<br><br> Let parmetis and ptsotch take edge weights and vertex weights<br><br> src/mat/partition/impls/pmetis/pmetis.c | 7 +++++++<br> src/mat/partition/impls/scotch/scotch.c | 6 +++---<br> 2 files changed, 10 insertions(+), 3 deletions(-)<br><div><br></div><div>> mpiexec -n 32 ./ex13 -cells 2,4,4, -dm_refine 5 -simplex 0 -dim 3 -potential_petscspace_degree 1 -potential_petscspace_order 1 -pc_type gamg -petscpartitioner_type simple -pc_gamg_repartition true -check_pointer_intensity 0</div></div></div></div></div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div>