[mpich-discuss] reading Ascii files using parallel I/o

[bob ilgner]

Have you considered using Hadoop ?

http://hadoop.apache.org/

Depending on much you want to scale this may be worth your while.
Otherwise I agree with mail below.



On Mon, May 23, 2011 at 3:55 PM, Brock Palen <brockp at umich.edu> wrote:
> Sreenivas,
>
> Note using binary IO tends to be faster as it takes less space for the same amount of data.
>
> You may also want to look at the hdf5/*-parallel modules on Nyx/Flux on campus. I find using hdf5 parallel IO to be simpler and more interoperable (modifiable and readable in matlab/idl with little effort) than invoking MPI-IO directly.
>
> Just my own $0.02 others with more parallel IO experience may have other input.
>
>
> Brock Palen
> www.umich.edu/~brockp
> Center for Advanced Computing
> brockp at umich.edu
> (734)936-1985
>
>
>
> On May 23, 2011, at 1:00 AM, Rajeev Thakur wrote:
>
>> MPI-IO doesn't support formatted I/O, i.e., the equivalent of fprintf and fscanf. However you can read/write unformatted character buffers.
>>
>> Rajeev
>>
>>
>>> Date: Sat, 21 May 2011 23:48:57 -0400
>>> From: <vvvnew at umich.edu>
>>> Subject: [mpich-discuss] reading Ascii files using parallel I/o
>>> To: mpich-discuss at mcs.anl.gov
>>> Message-ID: <7f8c0ee756294fb644ab193ff0338e1c at umich.edu>
>>> Content-Type: text/plain; charset="UTF-8"
>>>
>>>
>>> Hi,
>>> I want to know if there's a general way to read and write ASCII data
>>> using parallel input/output. I know there are only 3 Mpi data formats
>>> 'Native', 'Internal','External32', none of which support Ascii data. Since
>>> all my input data is in ASCII and i also need the output to be in ASCII to
>>> increase interoperability with other software, can someone suggest an
>>> efficient way to get around this problem. Right now iam sending all the
>>> data in and out through one node and makes the scaling pretty bad for large
>>> datasets, so parallel i/o with ascii would make my life a whole lot easier.
>>>
>>> Thanks
>>>
>>> Sreenivas
>>
>> _______________________________________________
>> mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>
>>
>
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>