<div dir="ltr">Just to clarify this a little more as it came up recently in some of Amal's thesis work using PyClaw. On a BlueGene/P the I/O nodes are available at some (usually fixed) density throughout the network. On the BlueGene/P system at KAUST, the majority of the system is available at the I/O node density of 128 compute nodes to 1 I/O node. If you use a non-MPIIO binary viewer, you are going to be limited by the network connection of a single I/O node to the parallel file system, which is in this case 10 Gb/s. The I/O nodes on BG/P are relatively low-powered ppc450 nodes, so it can be rather difficult for them to drive a full 10 Gb/s (1.25 GB/s), and they are limited as well by the collective network between the compute nodes and each I/O node (I believe this is 0.75 GB/s on our system). <div>
<br></div><div>The throughput performance of our parallel file system is limited by the file system controllers (each controller is backed by 8 servers), which have theoretical limits of 3GB/s each. We have a relatively low-powered system with 4 controllers, but that brings our theoretical I/O to 12 GB/s. When we do not enable MPIIO, we're limited to at most 1.25 GB/s, because we are only pushing on one of the I/O nodes, and we can theoretically achieve a maximum of 10% of the throughput capability of the file system. Switching on MPIIO eventually makes the I/O nodes no longer the theoretical bottleneck at 16 I/O nodes = 2048 compute nodes, or 2 racks of BG/P. </div>
<div><br>A</div><div><br><div class="gmail_quote">On Thu, Oct 13, 2011 at 5:05 AM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
On Oct 12, 2011, at 7:40 PM, Mohamad M. Nasr-Azadani wrote:<br>
<br>
> Hi again,<br>
><br>
> On the similar topic, how hard would it be to write a function similar to the PETSc's VecView() associated with Binary writer to do exactly the same thing, i.e. write a parallel vector into one single file but when writing all the processors are performing the task simultaneously?<br>
<br>
</div> Impossible. In the end in a standard filesystem the physical hard disk is connected to a single CPU and everything that gets written to that hard disk has to go through that one CPU; there is no way physically for a bunch of CPUs to write together onto a single physical disk.<br>
<br>
Now in high end parallel file systems each file may be spread over several hard disks (this is sometimes called stripping) (say 8). In that case there is some parallelism in writing the data since eight different parts of the vector can be sent through 8 different CPUs to 8 disks. But note that in general the number of these disks that file is spread over is small, like 8, it is never 10,000. When you have a parallel file system and use the option -viewer_binary_mpiio then the PETSc VecView() uses MPI IO to do the writing and you do get this level of parallelism so you may get slightly better performance than not using MPI IO.<br>
<br>
If you are seeing long wait times in VecView() with the binary file it is likely it is because the file server is connected via some pretty slow network to the actual compute nodes on parallel machine and nothing to do with the details of VecView(). You need to make sure you are writing directly to a disk that is on a compute node of the parallel machine, not over some network using NFS (Network File System), this can make a huge difference in time.<br>
<span class="HOEnZb"><font color="#888888"><br>
Barry<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
><br>
> Best,<br>
> Mohamad<br>
><br>
><br>
> On Wed, Oct 12, 2011 at 4:17 PM, Mohamad M. Nasr-Azadani <<a href="mailto:mmnasr@gmail.com">mmnasr@gmail.com</a>> wrote:<br>
> Thanks Barry. That makes perfect sense.<br>
><br>
> Best,<br>
> Mohamad<br>
><br>
><br>
> On Wed, Oct 12, 2011 at 3:50 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> On Oct 12, 2011, at 5:42 PM, Mohamad M. Nasr-Azadani wrote:<br>
><br>
> > Hi everyone,<br>
> ><br>
> > I think I know the answer to my question, but I was double checking.<br>
> > When using<br>
> > PetscViewerBinaryOpen();<br>
> ><br>
> > It is mentioned that<br>
> > "For writing files it only opens the file on processor 0 in the communicator."<br>
> ><br>
> > Does that mean when writing a parallel vector to file using VecView(), all the data from other processors is first sent to processor zero and then dumped into the file?<br>
><br>
> No all the data is not sent to process zero before writing. That is process 0 does not need enough memory to store all the data before writing.<br>
><br>
> Instead the processes take turns sending data to process 0 who immediately writes it out out to disk.<br>
><br>
> > If so, that would be a very slow processor for big datasets and large number of processor?<br>
><br>
> For less than a few thousand processes this is completely fine and nothing else would be much faster<br>
><br>
> > Any suggestions to speed that process up?<br>
><br>
> We have the various MPI IO options that uses MPI IO to have several processes writing to disks at the same time that is useful for very large numbers of processes.<br>
><br>
> Barry<br>
><br>
> ><br>
> > Best,<br>
> > Mohamad<br>
> ><br>
><br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div></div>