[mpich-discuss] reading Ascii files using parallel I/o

[vvvnew at umich.edu]

I havent heard of hadoop until now. Ill take a look at it.

Thanks


On Mon, 23 May 2011 16:12:16 +0200, bob ilgner <bobilgner at gmail.com> wrote:
> Have you considered using Hadoop ?
> 
> http://hadoop.apache.org/
> 
> Depending on much you want to scale this may be worth your while.
> Otherwise I agree with mail below.
> 
> 
> 
> On Mon, May 23, 2011 at 3:55 PM, Brock Palen <brockp at umich.edu> wrote:
>> Sreenivas,
>>
>> Note using binary IO tends to be faster as it takes less space for the
>> same amount of data.
>>
>> You may also want to look at the hdf5/*-parallel modules on Nyx/Flux on
>> campus. I find using hdf5 parallel IO to be simpler and more
>> interoperable (modifiable and readable in matlab/idl with little effort)
>> than invoking MPI-IO directly.
>>
>> Just my own $0.02 others with more parallel IO experience may have other
>> input.
>>
>>
>> Brock Palen
>> www.umich.edu/~brockp
>> Center for Advanced Computing
>> brockp at umich.edu
>> (734)936-1985
>>
>>
>>
>> On May 23, 2011, at 1:00 AM, Rajeev Thakur wrote:
>>
>>> MPI-IO doesn't support formatted I/O, i.e., the equivalent of fprintf
>>> and fscanf. However you can read/write unformatted character buffers.
>>>
>>> Rajeev
>>>
>>>
>>>> Date: Sat, 21 May 2011 23:48:57 -0400
>>>> From: <vvvnew at umich.edu>
>>>> Subject: [mpich-discuss] reading Ascii files using parallel I/o
>>>> To: mpich-discuss at mcs.anl.gov
>>>> Message-ID: <7f8c0ee756294fb644ab193ff0338e1c at umich.edu>
>>>> Content-Type: text/plain; charset="UTF-8"
>>>>
>>>>
>>>> Hi,
>>>> I want to know if there's a general way to read and write ASCII data
>>>> using parallel input/output. I know there are only 3 Mpi data formats
>>>> 'Native', 'Internal','External32', none of which support Ascii data.
>>>> Since
>>>> all my input data is in ASCII and i also need the output to be in
ASCII
>>>> to
>>>> increase interoperability with other software, can someone suggest an
>>>> efficient way to get around this problem. Right now iam sending all
the
>>>> data in and out through one node and makes the scaling pretty bad for
>>>> large
>>>> datasets, so parallel i/o with ascii would make my life a whole lot
>>>> easier.
>>>>
>>>> Thanks
>>>>
>>>> Sreenivas
>>>
>>> _______________________________________________
>>> mpich-discuss mailing list
>>> mpich-discuss at mcs.anl.gov
>>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>>
>>>
>>
>> _______________________________________________
>> mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss