[mpich-discuss] reading Ascii files using parallel I/o

[Brock Palen]

Haddop is an interesting option but requires a lot of infrastructure.

Sreenivas,

If you want to chat about what you are doing send an email into the CAC support address I send you in our internal ticket. We can setup a time to chat about options at Michigan.

Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp at umich.edu
(734)936-1985



On May 23, 2011, at 10:12 AM, bob ilgner wrote:

> Have you considered using Hadoop ?
> 
> http://hadoop.apache.org/
> 
> Depending on much you want to scale this may be worth your while.
> Otherwise I agree with mail below.
> 
> 
> 
> On Mon, May 23, 2011 at 3:55 PM, Brock Palen <brockp at umich.edu> wrote:
>> Sreenivas,
>> 
>> Note using binary IO tends to be faster as it takes less space for the same amount of data.
>> 
>> You may also want to look at the hdf5/*-parallel modules on Nyx/Flux on campus. I find using hdf5 parallel IO to be simpler and more interoperable (modifiable and readable in matlab/idl with little effort) than invoking MPI-IO directly.
>> 
>> Just my own $0.02 others with more parallel IO experience may have other input.
>> 
>> 
>> Brock Palen
>> www.umich.edu/~brockp
>> Center for Advanced Computing
>> brockp at umich.edu
>> (734)936-1985
>> 
>> 
>> 
>> On May 23, 2011, at 1:00 AM, Rajeev Thakur wrote:
>> 
>>> MPI-IO doesn't support formatted I/O, i.e., the equivalent of fprintf and fscanf. However you can read/write unformatted character buffers.
>>> 
>>> Rajeev
>>> 
>>> 
>>>> Date: Sat, 21 May 2011 23:48:57 -0400
>>>> From: <vvvnew at umich.edu>
>>>> Subject: [mpich-discuss] reading Ascii files using parallel I/o
>>>> To: mpich-discuss at mcs.anl.gov
>>>> Message-ID: <7f8c0ee756294fb644ab193ff0338e1c at umich.edu>
>>>> Content-Type: text/plain; charset="UTF-8"
>>>> 
>>>> 
>>>> Hi,
>>>> I want to know if there's a general way to read and write ASCII data
>>>> using parallel input/output. I know there are only 3 Mpi data formats
>>>> 'Native', 'Internal','External32', none of which support Ascii data. Since
>>>> all my input data is in ASCII and i also need the output to be in ASCII to
>>>> increase interoperability with other software, can someone suggest an
>>>> efficient way to get around this problem. Right now iam sending all the
>>>> data in and out through one node and makes the scaling pretty bad for large
>>>> datasets, so parallel i/o with ascii would make my life a whole lot easier.
>>>> 
>>>> Thanks
>>>> 
>>>> Sreenivas
>>> 
>>> _______________________________________________
>>> mpich-discuss mailing list
>>> mpich-discuss at mcs.anl.gov
>>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>> 
>>> 
>> 
>> _______________________________________________
>> mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>> 
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> 
>