[MPICH] Help With I/O

Wed Apr 18 17:45:28 CDT 2007

UPDATE: I tried to use gdb a little bit, running only one process, and it is throwing the exception on the first Write_shared call. But, I still don't know what the underlying problem is. I've re-arranged the code like we have discussed below:

#include "RecordRetrieval.h"#include "mpi.h"#include "time.h"#include "iostream.h"
void RecordRetrieval::QueryData(vector<bool> unencoded_vector, char * filename){    int num_processes;    int num_vector_elements;    float num_elements_per_rank;    int local_start_position;    int local_end_position;    char * buf;    int my_rank;    MPI::File input_file;    MPI::Status input_file_status;    MPI::File output_file;    MPI::Status output_file_status;    //MPI::Offset filesize;    char output_filename[30];    size_t i;    struct tm tim;    time_t now;    now = time(NULL);    tim = *(localtime(&now));    i = strftime(output_filename, 30, "%m_%d_%Y_%H_%M_%S", &tim);
    /* Let the system do what it needs to start up MPI */    MPI::Init();    /* Get my process rank */    my_rank = MPI::COMM_WORLD.Get_rank();    /* Find out how many processes are being used */    num_processes = MPI::COMM_WORLD.Get_size();    num_vector_elements = unencoded_vector.size();
    num_elements_per_rank = num_vector_elements / num_processes;    local_start_position = my_rank * (int)num_elements_per_rank;    if(my_rank == num_processes - 1)    {        if(num_elements_per_rank * num_processes == (int)num_elements_per_rank * num_processes)        {            local_end_position = local_start_position + ((int)num_elements_per_rank - 1);        }        else        {            local_end_position = (local_start_position + (int)num_elements_per_rank - 1) +                (((int)num_elements_per_rank * num_processes) - ((int)num_elements_per_rank * num_processes));        }    }    else    {        local_end_position = local_start_position + ((int)num_elements_per_rank - 1);    }
   buf = (char *) malloc(20); 
    input_file = MPI::File::Open(MPI::COMM_WORLD, filename, MPI::MODE_RDONLY,                    MPI::INFO_NULL);
    output_file = MPI::File::Open(MPI::COMM_WORLD, output_filename, MPI::MODE_CREATE | MPI::MODE_WRONLY, MPI::INFO_NULL);        // filesize = input_file.Get_size();    for(int i = local_start_position; i < local_end_position + 1; i++)    {        if(unencoded_vector[i])        {            input_file.Read_at(i * 20, buf, 20, MPI_BYTE, input_file_status);            output_file.Write_shared(buf, 20, MPI_BYTE, output_file_status);        }    }           free(buf);
    input_file.Close();    output_file.Close();                                              MPI::Finalize();}

From: matthew.chambers at vanderbilt.eduTo: erichpeterson at hotmail.com; mpich-discuss at mcs.anl.govSubject: RE: [MPICH] Help With I/ODate: Wed, 18 Apr 2007 14:36:43 -0500

The cout school of debugging has a lot to recommend it, but I don’t see the cout statements in your code and you haven’t described in more detail how you’ve used them to debug your for loop.  For example, how many iterations does the loop go through before crashing?  Does it even get through one?  I’m still not sure what your code is supposed to do.  Are you trying to read 20 bytes each iteration?  I would think that Read_at overwrites the contents of buf instead of appending onto the end of where the last read stopped.  In that case, you were right to have the output happen right after the input (however, it would probably be better to have a secondary buffer which buf is appended onto after each read operation, and after all the reads, you can write that secondary buffer out once.  It would be helpful if you would describe what your file reading is supposed to do more clearly.

From: owner-mpich-discuss at mcs.anl.gov [mailto:owner-mpich-discuss at mcs.anl.gov] On Behalf Of Erich PetersonSent: Wednesday, April 18, 2007 1:45 PMTo: mpich-discuss at mcs.anl.govSubject: RE: [MPICH] Help With I/O

Yeah you are totally right about the buf not being allocated, I think I have allocated it correctly now. I also think I should take the write function out of the for loop and place it at the end, and just let it write out what is in the buffer totally. Below is my revised code. It is still giving me the same errors though. Another problem is, I have no idea how to debug using Linux (I'm a .Net Windows guy), which is what I have to use for this project. I have like one week left of school, and I don't think I have enough time to learn gdb. So, I'm just been placing cout statements everywhere to see it I have correct file handles, etc...#include "RecordRetrieval.h"#include "mpi.h"#include "time.h"#include "iostream.h"void RecordRetrieval::QueryData(vector<bool> unencoded_vector, char * filename){    int num_processes;    int num_vector_elements;    float num_elements_per_rank;    int local_start_position;    int local_end_position;    char * buf;    int my_rank;    MPI::File input_file;    MPI::Status input_file_status;    MPI::File output_file;    MPI::Status output_file_status;    //MPI::Offset filesize;    char output_filename[30];    size_t i;    struct tm tim;    time_t now;    now = time(NULL);    tim = *(localtime(&now));    i = strftime(output_filename, 30, "%m_%d_%Y_%H_%M_%S", &tim);    /* Let the system do what it needs to start up MPI */    MPI::Init();    /* Get my process rank */    my_rank = MPI::COMM_WORLD.Get_rank();    /* Find out how many processes are being used */    num_processes = MPI::COMM_WORLD.Get_size();    num_vector_elements = unencoded_vector.size();    num_elements_per_rank = num_vector_elements / num_processes;    local_start_position = my_rank * (int)num_elements_per_rank;    if(my_rank == num_processes - 1)    {        if(num_elements_per_rank * num_processes == (int)num_elements_per_rank * num_processes)        {            local_end_position = local_start_position + ((int)num_elements_per_rank - 1);        }        else        {            local_end_position = (local_start_position + (int)num_elements_per_rank - 1) +                (((int)num_elements_per_rank * num_processes) - ((int)num_elements_per_rank * num_processes));        }    }    else    {        local_end_position = local_start_position + ((int)num_elements_per_rank - 1);    }   buf = (char *) malloc((local_end_position - local_start_position) + 1);     input_file = MPI::File::Open(MPI::COMM_WORLD, filename, MPI::MODE_RDONLY,                    MPI::INFO_NULL);    output_file = MPI::File::Open(MPI::COMM_WORLD, output_filename, MPI::MODE_CREATE | MPI::MODE_WRONLY, MPI::INFO_NULL);        // filesize = input_file.Get_size();    for(int i = local_start_position; i < local_end_position + 1; i++)    {        if(unencoded_vector[i])        {            input_file.Read_at(i * 20, buf, 20, MPI_CHAR, input_file_status);        }    }        output_file.Write_shared(buf, ((local_end_position - local_start_position) + 1), MPI_CHAR, output_file_status);       free(buf);    input_file.Close();    output_file.Close();                                              MPI::Finalize();}
_________________________________________________________________
Discover the new Windows Vista
http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20070418/1d09d6bc/attachment.htm>