MPIIO is a parallel file reader. Your application should output in the PETSc binary format (it's easy) for the best performance, an ASCII file cannot be brought in quickly using any existing code. Need to dash, there are MATLAB code examples in bin/ that show how to write out the binary format.<div>
<br></div><div>A<br><br><div class="gmail_quote">On Sun, Jan 9, 2011 at 4:45 AM, Zuhair Khayyat <span dir="ltr"><<a href="mailto:zuhair.khayyat@kaust.edu.sa">zuhair.khayyat@kaust.edu.sa</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div dir="ltr">Dear Aron,<br><br>Actually the file is an output of another application, and is stored in a simple text file. I have read about MPIIO, however couldn't figure out if it is a parallel file reader or not.<br>
<br>Have you ever tried or worked on a parallel file reader to a common global matrix in PETSc, where each processor has part of the file? Thank you<br><br>Regards,<br><font color="#888888">Zuhair Khayyat</font><div><div>
</div><div class="h5"><br><br><div class="gmail_quote">On Sat, Jan 8, 2011 at 8:42 PM, Aron Ahmadia <span dir="ltr"><<a href="mailto:aron.ahmadia@kaust.edu.sa" target="_blank">aron.ahmadia@kaust.edu.sa</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left:1px solid rgb(204, 204, 204);margin:0pt 0pt 0pt 0.8ex;padding-left:1ex">if the file is stored in the PETSc format, you can use PETSc to pull the file in using MPIIO, which should be (hopefully) faster.<div>
<br></div><div><a href="http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Viewer/PetscViewerBinarySetMPIIO.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Viewer/PetscViewerBinarySetMPIIO.html</a></div>
<div><br></div><div>Feel free to pop these questions to petsc-users (cc'd), there are more of them and they usually know more than me :)</div><div><br></div><div><font color="#888888">-Aron</font><div><div></div><div>
<br><br><div class="gmail_quote">On Sat, Jan 8, 2011 at 2:21 PM, Zuhair Khayyat <span dir="ltr"><<a href="mailto:zuhair.khayyat@kaust.edu.sa" target="_blank">zuhair.khayyat@kaust.edu.sa</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left:1px solid rgb(204, 204, 204);margin:0pt 0pt 0pt 0.8ex;padding-left:1ex"><div dir="ltr">Dear Aron,<br><br>Currently I am using PETSc for my research to implement a graph mining algorithm on a large cluster, and I would like ask you some questions due to your experience with this tool.<br>
<br>Have you ever tried to optimize allocating a very large matrix from input file in parallel? I have a very large file (around 10 GB) and it takes too long to allocate the matrix through the main node. I am planning to implement a distributed parallel file reader that split the original file and make each node reads it separately into a common matrix.<br>
<br>Is there and other tools that are comparable to PETSc in which they have parallel file reader?<br><br>Thank you for your help<br><br>Regards,<br>Zuhair Khayyat<br></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>