Gus, Wei and all,<div><br></div><div>Thanks very much for your answer. It clarified lots of things and now I think I can really choose the right way to move.</div><div><br></div><div>My case is like the second Gus described: I have a big file with some initial data and just part of it really need to go to each respective process. Also, this could be done while the computation is taking place. It is better to mix computation and communication instead of concentrate all communication in just one step (when talking about large data), right?<br>
<div><br></div><div>Wei also told me that I could use <span class="Apple-style-span" style="border-collapse: collapse; ">MPI_File_read_all(). As I said, I can read parts of the file during my steps. I will look for some collective function to do it and also try what you said.</span></div>
<div><span class="Apple-style-span" style="border-collapse: collapse;"><br></span></div><div><span class="Apple-style-span" style="border-collapse: collapse;">One question: nowadays my Master process also participates in the computation and this is particularly fine because all processes run the same code. However, I have thought about create a special process to work like an "data server", responsible to "broadcast" this information along the other processes. Is there some reason that might jeopardize this implementation?</span></div>
<div><span class="Apple-style-span" style="border-collapse: collapse;"><br></span></div><div><span class="Apple-style-span" style="border-collapse: collapse;">Thanks in advance,</span></div><div><span class="Apple-style-span" style="border-collapse: collapse;">Luiz</span></div>
<div><br><div class="gmail_quote">On Wed, Oct 22, 2008 at 1:50 PM, Gus Correa <span dir="ltr"><<a href="mailto:gus@ldeo.columbia.edu">gus@ldeo.columbia.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hello Kamaraju, Luiz and list<br>
<br>
Reading on processs 0 and broadcasting to all others is typically<br>
what is done in most programs we use here (ocean, atmosphere, climate models).<br>
For control files, with namelists for instance, which are small but contain global data<br>
that is needed by all processes, this works very well.<br>
<br>
For larger files, e.g., binaries with initial conditions data on the global grid,<br>
this is often done also, but with a twist, not through broadcast.<br>
These are domain decomposition applications, where each process works on a<br>
subdomain, and hence needs to know only the part of the data in one subdomain<br>
(and exchange domain boundaries with neighbor domains/processes as the solution is marched in time).<br>
Hence. for this type of array/grid data, there is no need to broadcast the global array,<br>
but to scatter the data from the global domain to the subdomains/processes.<br>
Often times even the subdomain data is quite large (say 3D arrays), and need to be split<br>
into smaller chunks (say 2D slices), to keep message size manageable.<br>
In this case you scatter the smaller chunks in a loop.<br>
MPI types are often used to organize the data structures being exchanged,<br>
decomposing the data on one process and reassembling them on another process, etc.<br>
<br>
This technique is very traditional, the "master-slave" model,<br>
and precedes the MPI I/O functions, parallel file systems, etc.<br>
The mirror image of it is gathering the data from all subdomains by process 0,<br>
which very often is the one responsible to write the data to output files.<br>
Indeed, this serializes the computation to some extent,<br>
but it is safe for I/O in systems where, say, NFS may become a bottleneck.<br>
(If you use local disks on the nodes of a cluster this is not a problem, but then you need to<br>
take care of staging in data and staging out the results to/from the local disks,<br>
and post-processing the files perhaps.)<br>
If I have, say, 64 processes banging together on a single NFS server things typically break down.<br>
If I use MPI to gather/scatter big data arrays, and funnel the I/O trough process 0, NFS doesn't suffer.<br>
Old fashioned but functional. :)<br>
<br>
Another list subscriber pointed out a different solution, using the MPI I/O functions.<br>
I believe it works well, say, for raw binary files, but we use more structured stuff here (e.g. NetCDF format).<br>
<br>
I hope this helps,<br>
Gus Correa<br><font color="#888888">
<br>
-- <br>
---------------------------------------------------------------------<br>
Gustavo J. Ponce Correa, PhD - Email: <a href="mailto:gus@ldeo.columbia.edu" target="_blank">gus@ldeo.columbia.edu</a><br>
Lamont-Doherty Earth Observatory - Columbia University<br>
P.O. Box 1000 [61 Route 9W] - Palisades, NY, 10964-8000 - USA<br>
---------------------------------------------------------------------</font><div class="Ih2E3d"><br>
<br>
<br>
Luiz Carlos da Costa Junior wrote:<br>
<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="Ih2E3d">
Hi all,<br>
<br>
Let me join this conversation. I also "suffer" from these doubts. In my case, I have an application in two versions, Windows (NTFS) and Linux (FAT32) and I have first implemented the first approach (make one separated copy for each machine).<br>
<br>
But recently, I started to deal with bigger files (200Mb ~ 1Gb) and this became very inefficiently. Actually, the reason I suspect is that even we have multiple processes, the hard disk device that is responsible for manage all these readings is just one. In other words, this operation is intrinsically sequential and became a bottleneck (am I right?).<br>
<br>
I didn't changed my implementation yet, but I was thinking to move to the second approach (rank 0 reads and BCast the info) expecting to have better results.<br>
<br>
Does anyone have any experience?<br>
<br>
Actually I am not sure if this will be better. I understand that MPI uses sockets to pass all messages and an natural question is if this operation is faster than reading from files?<br>
<br>
Best regards,<br>
Luiz<br>
<br></div><div class="Ih2E3d">
On Wed, Oct 22, 2008 at 12:10 AM, Rajeev Thakur <<a href="mailto:thakur@mcs.anl.gov" target="_blank">thakur@mcs.anl.gov</a> <mailto:<a href="mailto:thakur@mcs.anl.gov" target="_blank">thakur@mcs.anl.gov</a>>> wrote:<br>
<br>
How big is the file? What kind of file system is it on?<br>
<br>
Rajeev<br>
<br>
> -----Original Message-----<br>
> From: <a href="mailto:owner-mpich-discuss@mcs.anl.gov" target="_blank">owner-mpich-discuss@mcs.anl.gov</a><br>
<mailto:<a href="mailto:owner-mpich-discuss@mcs.anl.gov" target="_blank">owner-mpich-discuss@mcs.anl.gov</a>><br>
> [mailto:<a href="mailto:owner-mpich-discuss@mcs.anl.gov" target="_blank">owner-mpich-discuss@mcs.anl.gov</a><br>
<mailto:<a href="mailto:owner-mpich-discuss@mcs.anl.gov" target="_blank">owner-mpich-discuss@mcs.anl.gov</a>>] On Behalf Of<br>
> Kamaraju Kusumanchi<br>
> Sent: Tuesday, October 21, 2008 8:27 PM<br></div><div class="Ih2E3d">
> To: <a href="mailto:mpich-discuss@mcs.anl.gov" target="_blank">mpich-discuss@mcs.anl.gov</a> <mailto:<a href="mailto:mpich-discuss@mcs.anl.gov" target="_blank">mpich-discuss@mcs.anl.gov</a>><br>
> Subject: [mpich-discuss] read the same file on all processes<br>
><br>
> Hi all,<br>
><br>
> I have a file which needs to be read on all the processes<br>
> of an MPI job. If I read the same file simultaneously on all<br>
> the processes, will it cause any problems?<br>
><br>
> I can think of two other options such as<br>
><br>
> - make multiple copies of the same file and read a separate<br>
> file on different processes<br>
> - read the file on rank 0 process, then use MPI_Bcast and<br>
> transfer the contents across the remaining processes.<br>
><br>
> Which approach should be preferred? I am thinking this<br>
> must be something encountered by others. So, if there is a<br>
> book/web page which explains these kind of things, a pointer<br>
> to them would be most appreciated.<br>
><br>
> regards<br>
> raju<br>
><br>
><br>
<br>
<br>
</div></blockquote>
<br>
</blockquote></div><br></div></div>