<div dir="ltr">Hi all,<div><br></div><div> I'm trying to do basic binary input/output for parallel matrices. Following the example in "<a href="https://github.com/erdc-cm/petsc4py/blob/master/demo/binary-io/matvecio.py">petsc4py/demo/binary-io/matvecio.py</a> ", the code below seems to work fine (raw python file attached) with "mpirun -np 2 python ./test_binwrite.py" until I uncomment "#A.setType(PETSc.Mat.Type.DENSE)" in order to do it for parallel dense matrices. Then it crashes with error (stderr file attached, see below for highlighted message). What am I doing wrong?</div><div><br></div><div>Python Code</div><div><div><font color="#696969">========================================================================================</font></div></div><div><span style="color:rgb(105,105,105)"><br></span></div><div><span style="color:rgb(105,105,105)">#!/usr/bin/env python</span><br></div><pre style="background:rgb(255,255,255)"><span style="color:rgb(105,105,105)">"""</span><font color="#000000">
</font><span style="color:rgb(105,105,105)">Created on Wed Jun 29 22:05:59 2016</span><font color="#000000">
</font><span style="color:rgb(105,105,105)"></span><font color="#000000">
</font><span style="color:rgb(105,105,105)">@author: </span><span style="color:rgb(113,68,196)"><a href="mailto:daneel@utexas.edu">daneel@utexas.edu</a></span><span style="color:rgb(105,105,105)"></span><font color="#000000">
</font><span style="color:rgb(105,105,105)">"""</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">import</span><font color="#000000"> numpy </font><span style="color:rgb(128,0,0);font-weight:bold">as</span><font color="#000000"> np
</font><span style="color:rgb(128,0,0);font-weight:bold">from</span><font color="#000000"> petsc4py </font><span style="color:rgb(128,0,0);font-weight:bold">import</span><font color="#000000"> PETSc
</font><span style="color:rgb(128,0,0);font-weight:bold">Print</span><font color="#000000"> </font><span style="color:rgb(128,128,48)">=</span><font color="#000000"> PETSc</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">Sys</font><span style="color:rgb(128,128,48)">.</span><span style="color:rgb(128,0,0);font-weight:bold">Print</span><font color="#000000">
matsize </font><span style="color:rgb(128,128,48)">=</span><font color="#000000"> </font><span style="color:rgb(0,140,0)">100</span><font color="#000000">
filename </font><span style="color:rgb(128,128,48)">=</span><font color="#000000"> </font><span style="color:rgb(0,0,230)">'tmpMatrix-A.dat'</span><font color="#000000">
A </font><span style="color:rgb(128,128,48)">=</span><font color="#000000"> PETSc</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">Mat</font><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,128,48)">.</span><font color="#000000">create</font><span style="color:rgb(128,128,48)">(</span><font color="#000000">PETSc</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">COMM_WORLD</font><span style="color:rgb(128,128,48)">)</span><font color="#000000">
A</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">setSizes</font><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">[</span><font color="#000000">matsize</font><span style="color:rgb(128,128,48)">,</span><font color="#000000"> matsize</font><span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,128,48)">)</span><font color="#000000">
</font><span style="color:rgb(105,105,105)">#A.setType(PETSc.Mat.Type.DENSE)</span><font color="#000000">
A</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">setUp</font><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><font color="#000000">
Istart</font><span style="color:rgb(128,128,48)">,</span><font color="#000000"> Iend </font><span style="color:rgb(128,128,48)">=</span><font color="#000000"> A</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">getOwnershipRange</font><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">for</span><font color="#000000"> I </font><span style="color:rgb(128,0,0);font-weight:bold">in</span><font color="#000000"> </font><span style="color:rgb(64,0,0)">xrange</span><span style="color:rgb(128,128,48)">(</span><font color="#000000">Istart</font><span style="color:rgb(128,128,48)">,</span><font color="#000000"> Iend</font><span style="color:rgb(128,128,48)">)</span><font color="#000000"> </font><span style="color:rgb(128,128,48)">:</span><font color="#000000">
A</font><span style="color:rgb(128,128,48)">[</span><font color="#000000">I</font><span style="color:rgb(128,128,48)">,</span><span style="color:rgb(128,128,48)">:</span><span style="color:rgb(128,128,48)">]</span><font color="#000000"> </font><span style="color:rgb(128,128,48)">=</span><font color="#000000"> np</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">random</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">random</font><span style="color:rgb(128,128,48)">(</span><font color="#000000">matsize</font><span style="color:rgb(128,128,48)">)</span><font color="#000000">
A</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">assemble</font><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">Print</span><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(0,0,230)">"Random matrix assembled. Now writing to temporary file ..."</span><span style="color:rgb(128,128,48)">)</span><font color="#000000">
</font><span style="color:rgb(105,105,105)"># save</span><font color="#000000">
viewer </font><span style="color:rgb(128,128,48)">=</span><font color="#000000"> PETSc</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">Viewer</font><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,128,48)">.</span><font color="#000000">createBinary</font><span style="color:rgb(128,128,48)">(</span><font color="#000000">filename</font><span style="color:rgb(128,128,48)">,</span><font color="#000000"> </font><span style="color:rgb(0,0,230)">'w'</span><span style="color:rgb(128,128,48)">)</span><font color="#000000">
viewer</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">pushFormat</font><span style="color:rgb(128,128,48)">(</span><font color="#000000">viewer</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">Format</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">NATIVE</font><span style="color:rgb(128,128,48)">)</span><font color="#000000">
viewer</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">view</font><span style="color:rgb(128,128,48)">(</span><font color="#000000">A</font><span style="color:rgb(128,128,48)">)</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">Print</span><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(0,0,230)">"Written to "</span><span style="color:rgb(128,128,48)">,</span><font color="#000000"> filename</font><span style="color:rgb(128,128,48)">)</span><font color="#000000">
</font><span style="color:rgb(105,105,105)"># reload</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">Print</span><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(0,0,230)">"Now reloading from "</span><span style="color:rgb(128,128,48)">,</span><font color="#000000"> filename</font><span style="color:rgb(128,128,48)">)</span><font color="#000000">
viewer_new </font><span style="color:rgb(128,128,48)">=</span><font color="#000000"> PETSc</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">Viewer</font><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,128,48)">.</span><font color="#000000">createBinary</font><span style="color:rgb(128,128,48)">(</span><font color="#000000">filename</font><span style="color:rgb(128,128,48)">,</span><font color="#000000"> </font><span style="color:rgb(0,0,230)">'r'</span><span style="color:rgb(128,128,48)">)</span><font color="#000000">
viewer_new</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">pushFormat</font><span style="color:rgb(128,128,48)">(</span><font color="#000000">viewer</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">getFormat</font><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,128,48)">)</span><font color="#000000">
A_new </font><span style="color:rgb(128,128,48)">=</span><font color="#000000"> PETSc</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">Mat</font><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,128,48)">.</span><font color="#000000">load</font><span style="color:rgb(128,128,48)">(</span><font color="#000000">viewer_new</font><span style="color:rgb(128,128,48)">)</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">assert</span><font color="#000000"> A_new</font><span style="color:rgb(128,128,48)">.</span><font color="#000000">equal</font><span style="color:rgb(128,128,48)">(</span><font color="#000000">A</font><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,128,48)">,</span><font color="#000000"> </font><span style="color:rgb(0,0,230)">"Reload unsuccessful"</span><font color="#000000">
</font><span style="color:rgb(128,0,0);font-weight:bold">Print</span><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(0,0,230)">"Reload successful"</span><span style="color:rgb(128,128,48)">)</span><font color="#000000">
</font></pre>
<pre style="background:rgb(255,255,255)"><font color="#000000">========================================================================================
</font><span style="color:rgb(0,0,0)">
</span></pre><pre style="color:rgb(0,0,0);background:rgb(255,255,255)"><br></pre><pre style="color:rgb(0,0,0);background:rgb(255,255,255)">stderr output for parallel dense matrices:</pre><pre style="background:rgb(255,255,255)"><font color="#000000">========================================================================================</font><span style="color:rgb(0,0,0)">
</span></pre><pre style="background:rgb(255,255,255)"><span style="color:rgb(0,0,0)">Random matrix assembled<span style="color:rgb(128,0,0);font-weight:bold">.</span> Now writing to temporary file <span style="color:rgb(128,0,0);font-weight:bold">.</span><span style="color:rgb(128,0,0);font-weight:bold">.</span><span style="color:rgb(128,0,0);font-weight:bold">.</span>
Written to tmpMatrix-A<span style="color:rgb(128,0,0);font-weight:bold">.</span>dat
Now reloading from tmpMatrix-A<span style="color:rgb(128,0,0);font-weight:bold">.</span>dat
Traceback <span style="color:rgb(128,0,128)">(</span>most recent call last<span style="color:rgb(128,0,128)">)</span><span style="color:rgb(128,128,48)">:</span>
File <span style="color:rgb(0,0,230)">".</span><span style="color:rgb(64,1,90)">/test_binwrite.py</span><span style="color:rgb(0,0,230)">"</span>, line <span style="color:rgb(0,140,0)">36</span>, <span style="color:rgb(128,0,0);font-weight:bold">in</span> <span style="color:rgb(227,74,220)"><</span>module<span style="color:rgb(227,74,220)">></span>
Traceback <span style="color:rgb(128,0,128)">(</span>most recent call last<span style="color:rgb(128,0,128)">)</span><span style="color:rgb(128,128,48)">:</span>
A_new <span style="color:rgb(68,170,221)">=</span> PETSc<span style="color:rgb(128,0,0);font-weight:bold">.</span>Mat<span style="color:rgb(128,0,128)">(</span><span style="color:rgb(128,0,128)">)</span><span style="color:rgb(128,0,0);font-weight:bold">.</span>load<span style="color:rgb(128,0,128)">(</span>viewer_new<span style="color:rgb(128,0,128)">)</span>
File <span style="color:rgb(0,0,230)">".</span><span style="color:rgb(64,1,90)">/test_binwrite.py</span><span style="color:rgb(0,0,230)">"</span>, line <span style="color:rgb(0,140,0)">36</span>, <span style="color:rgb(128,0,0);font-weight:bold">in</span> <span style="color:rgb(227,74,220)"><</span>module<span style="color:rgb(227,74,220)">></span>
File <span style="color:rgb(0,0,230)">"</span><span style="color:rgb(64,1,90)">PETSc/Mat.pyx</span><span style="color:rgb(0,0,230)">"</span>, line <span style="color:rgb(0,140,0)">642</span>, <span style="color:rgb(128,0,0);font-weight:bold">in</span> petsc4py<span style="color:rgb(128,0,0);font-weight:bold">.</span>PETSc<span style="color:rgb(128,0,0);font-weight:bold">.</span>Mat<span style="color:rgb(128,0,0);font-weight:bold">.</span>load <span style="color:rgb(128,0,128)">(</span><span style="color:rgb(64,1,90)">src/petsc4py.PETSc.c</span><span style="color:rgb(128,128,48)">:</span><span style="color:rgb(0,140,0)">118243</span><span style="color:rgb(128,0,128)">)</span>
A_new <span style="color:rgb(68,170,221)">=</span> PETSc<span style="color:rgb(128,0,0);font-weight:bold">.</span>Mat<span style="color:rgb(128,0,128)">(</span><span style="color:rgb(128,0,128)">)</span><span style="color:rgb(128,0,0);font-weight:bold">.</span>load<span style="color:rgb(128,0,128)">(</span>viewer_new<span style="color:rgb(128,0,128)">)</span>
File <span style="color:rgb(0,0,230)">"</span><span style="color:rgb(64,1,90)">PETSc/Mat.pyx</span><span style="color:rgb(0,0,230)">"</span>, line <span style="color:rgb(0,140,0)">642</span>, <span style="color:rgb(128,0,0);font-weight:bold">in</span> petsc4py<span style="color:rgb(128,0,0);font-weight:bold">.</span>PETSc<span style="color:rgb(128,0,0);font-weight:bold">.</span>Mat<span style="color:rgb(128,0,0);font-weight:bold">.</span>load <span style="color:rgb(128,0,128)">(</span><span style="color:rgb(64,1,90)">src/petsc4py.PETSc.c</span><span style="color:rgb(128,128,48)">:</span><span style="color:rgb(0,140,0)">118243</span><span style="color:rgb(128,0,128)">)</span>
petsc4py<span style="color:rgb(128,0,0);font-weight:bold">.</span>PETSc<span style="color:rgb(128,0,0);font-weight:bold">.</span>Errorpetsc4py<span style="color:rgb(128,0,0);font-weight:bold">.</span>PETSc<span style="color:rgb(128,0,0);font-weight:bold">.</span>Error<span style="color:rgb(128,128,48)">:</span> <span style="color:rgb(128,128,48)">:</span> error code <span style="color:rgb(0,140,0)">55</span>
<span style="color:rgb(128,128,48)">[</span><span style="color:rgb(0,0,230)">0</span><span style="color:rgb(128,128,48)">]</span> MatLoad<span style="color:rgb(128,0,128)">(</span><span style="color:rgb(128,0,128)">)</span> line <span style="color:rgb(0,140,0)">1013</span> <span style="color:rgb(128,0,0);font-weight:bold">in</span> <span style="color:rgb(64,1,90)">/home/daneel</span><span style="color:rgb(64,1,90)">/</span><span style="color:rgb(128,0,0);font-weight:bold">.</span><span style="color:rgb(64,1,90)">local/src/petsc-3.7.2/src/mat/interface/matrix.c</span>
<span style="color:rgb(128,128,48)">[</span><span style="color:rgb(0,0,230)">0</span><span style="color:rgb(128,128,48)">]</span> MatLoad_MPIAIJ<span style="color:rgb(128,0,128)">(</span><span style="color:rgb(128,0,128)">)</span> line <span style="color:rgb(0,140,0)">2947</span> <span style="color:rgb(128,0,0);font-weight:bold">in</span> <span style="color:rgb(64,1,90)">/home/daneel</span><span style="color:rgb(64,1,90)">/</span><span style="color:rgb(128,0,0);font-weight:bold">.</span><span style="color:rgb(64,1,90)">local/src/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</span>
<span style="color:rgb(128,128,48)">[</span><span style="color:rgb(0,0,230)">0</span><span style="color:rgb(128,128,48)">]</span> MatLoad_MPIAIJ<span style="color:rgb(128,0,128)">(</span><span style="color:rgb(128,0,128)">)</span> line <span style="color:rgb(0,140,0)">2947</span> <span style="color:rgb(128,0,0);font-weight:bold">in</span> <span style="color:rgb(64,1,90)">/home/daneel</span><span style="color:rgb(64,1,90)">/</span><span style="color:rgb(128,0,0);font-weight:bold">.</span><span style="color:rgb(64,1,90)">local/src/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</span>
<span style="color:rgb(128,128,48)">[</span><span style="color:rgb(0,0,230)">0</span><span style="color:rgb(128,128,48)">]</span> Out of memory<span style="color:rgb(128,0,0);font-weight:bold">.</span> Allocated<span style="color:rgb(128,128,48)">:</span> <span style="color:rgb(0,140,0)">0</span>, Used by process<span style="color:rgb(128,128,48)">:</span> <span style="color:rgb(0,140,0)">23687168</span>
<span style="color:rgb(128,128,48)">[</span><span style="color:rgb(0,0,230)">0</span><span style="color:rgb(128,128,48)">]</span> Memory requested <span style="color:rgb(0,140,0)">5942180016</span>
error code <span style="color:rgb(0,140,0)">55</span>
<span style="color:rgb(128,128,48)">[</span><span style="color:rgb(0,0,230)">1</span><span style="color:rgb(128,128,48)">]</span> MatLoad<span style="color:rgb(128,0,128)">(</span><span style="color:rgb(128,0,128)">)</span> line <span style="color:rgb(0,140,0)">1013</span> <span style="color:rgb(128,0,0);font-weight:bold">in</span> <span style="color:rgb(64,1,90)">/home/daneel</span><span style="color:rgb(64,1,90)">/</span><span style="color:rgb(128,0,0);font-weight:bold">.</span><span style="color:rgb(64,1,90)">local/src/petsc-3.7.2/src/mat/interface/matrix.c</span>
<span style="color:rgb(128,128,48)">[</span><span style="color:rgb(0,0,230)">1</span><span style="color:rgb(128,128,48)">]</span> MatLoad_MPIAIJ<span style="color:rgb(128,0,128)">(</span><span style="color:rgb(128,0,128)">)</span> line <span style="color:rgb(0,140,0)">2967</span> <span style="color:rgb(128,0,0);font-weight:bold">in</span> <span style="color:rgb(64,1,90)">/home/daneel</span><span style="color:rgb(64,1,90)">/</span><span style="color:rgb(128,0,0);font-weight:bold">.</span><span style="color:rgb(64,1,90)">local/src/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</span>
<span style="color:rgb(128,128,48)">[</span><span style="color:rgb(0,0,230)">1</span><span style="color:rgb(128,128,48)">]</span> MatLoad_MPIAIJ<span style="color:rgb(128,0,128)">(</span><span style="color:rgb(128,0,128)">)</span> line <span style="color:rgb(0,140,0)">2967</span> <span style="color:rgb(128,0,0);font-weight:bold">in</span> <span style="color:rgb(64,1,90)">/home/daneel</span><span style="color:rgb(64,1,90)">/</span><span style="color:rgb(128,0,0);font-weight:bold">.</span><span style="color:rgb(64,1,90)">local/src/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</span>
<span style="color:rgb(128,128,48)">[</span><span style="color:rgb(0,0,230)">1</span><span style="color:rgb(128,128,48)">]</span> Out of memory<span style="color:rgb(128,0,0);font-weight:bold">.</span> Allocated<span style="color:rgb(128,128,48)">:</span> <span style="color:rgb(0,140,0)">0</span>, Used by process<span style="color:rgb(128,128,48)">:</span> <span style="color:rgb(0,140,0)">23506944</span>
<span style="color:rgb(128,128,48)">[</span><span style="color:rgb(0,0,230)">1</span><span style="color:rgb(128,128,48)">]</span> Memory requested <span style="color:rgb(0,140,0)">18446744072415633408</span>
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation<span style="color:rgb(128,0,0);font-weight:bold">.</span>
--------------------------------------------------------------------------</span><font face="arial, sans-serif"><span style="white-space:normal">
</span></font></pre>
<div><div><div>========================================================================================</div><div><br></div></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>---<br>Analabha Roy<br>Postdoctoral Fellow<br><a href="http://www.nithep.ac.za/" target="_blank">National Institute of Theoretical Physics (NiTheP)</a><br>Private Bag X1, Matieland,<br>Stellenbosch, South Africa,7602<br>Emails: <a href="mailto:daneel@utexas.edu" target="_blank">daneel@utexas.edu</a>, <a href="mailto:daneel@sun.ac.za" target="_blank">daneel@sun.ac.za</a>, <a href="mailto:hariseldon99@gmail.com" target="_blank">hariseldon99@gmail.com</a><br><div><font face="tahoma, sans-serif">Webpage: <a href="http://www.ph.utexas.edu/~daneel/" target="_blank">http://www.ph.utexas.edu/~daneel/</a></font></div></div></div></div>
</div></div>