<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
We use petsc4py as a solver suite in our [FiPy](<a href="https://www.ctcms.nist.gov/fipy" class="">https://www.ctcms.nist.gov/fipy</a>) Python-based PDE solver package. Some time back, I refactored some of the code and provoked a deadlock situation in our test
suite. I have been tearing what remains of my hair out trying to isolate things and am at a loss. I’ve gone through the refactoring line-by-line and I just don’t think I’ve changed anything substantive, just how the code is organized.
<div class=""><br class="">
</div>
<div class="">I have posted a branch that exhibits the issue at <a href="https://github.com/usnistgov/fipy/pull/761" class="">https://github.com/usnistgov/fipy/pull/761</a></div>
<div class=""><br class="">
</div>
<div class="">I explain in greater detail in that “pull request” how to reproduce, but in short, after a substantial number of our tests run, the code either deadlocks or raises exceptions:</div>
<div class=""><br class="">
</div>
<div class="">On processor 0 in </div>
<div class=""><br class="">
</div>
<div class=""> matrix.setUp() </div>
<div class=""><br class="">
</div>
<div class="">specifically in</div>
<div class=""><br class="">
</div>
<div class=""> <span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">[</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">0</span><span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">]
</span><span class="pl-v" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(227, 98, 9);">PetscSplitOwnership</span><span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">()
</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">line</span><span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">
</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">93</span><span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">
</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">in</span><span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">
</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-v" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(227, 98, 9);">Users</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">runner</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">miniforge3</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">conda</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">-</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">bld</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">petsc_1601473259434</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">work</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">src</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">sys</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">utils</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">psplit</span><span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">.</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">c</span></div>
<div class=""><br class="">
</div>
<div class="">and on other processors a few lines earlier in</div>
<div class=""><br class="">
</div>
<div class=""> matrix.create(comm)</div>
<div class=""><br class="">
</div>
<div class="">specifically in </div>
<div class=""><br class="">
</div>
<div class=""> <span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">[</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">1</span><span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">]
</span><span class="pl-v" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(227, 98, 9);">PetscCommDuplicate</span><span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">()
</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">line</span><span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">
</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">126</span><span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">
</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">in</span><span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">
</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-v" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(227, 98, 9);">Users</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">runner</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">miniforge3</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">conda</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">-</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">bld</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">petsc_1601473259434</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">work</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">src</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">sys</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">objects</span><span class="pl-c1" style="font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box; color: rgb(0, 92, 197);">/</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">tagm</span><span style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2;" class="">.</span><span class="pl-s1" style="color: rgb(36, 41, 46); font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 11.9px; orphans: 2; widows: 2; box-sizing: border-box;">c</span></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">The circumstances that lead to this failure are really fragile and it seems likely due to some memory corruption. Particularly likely given that I can make the failure go away by removing seemingly irrelevant things like</div>
<div class=""><br class="">
</div>
<div class=""> >>> from scipy.stats.mstats import argstoarray</div>
<div class=""><br class="">
</div>
<div class="">Note that when I run the full test suite after taking out this scipy import, the same problem just arises elsewhere without any obvious similar import trigger.</div>
<div class=""><br class="">
</div>
<div class="">Running with `-malloc_debug true` doesn’t illuminate anything.</div>
<div class=""><br class="">
</div>
<div class="">I’ve run with `-info` and `-log_trace` and don’t see any obvious issues, but there’s a ton of output.</div>
<div class="">
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I have tried reducing things to a minimal reproducible example, but unfortunately things remain way too complicated and idiosyncratic to FiPy. I’m grateful for any help anybody can offer despite the mess that I’m offering.</div>
</div>
</body>
</html>