[Arvid Bessen]

On Jan 12, 2007, at 1:51 PM, Matthew Knepley wrote:

> Why are you creating that ISGeneral with PETSC_COMM_WORLD. It does not
> seem to be global? Shouldn't it be PETSC_COMM_SELF? Also, what are you
> trying to do?
>
Thanks, that's a very good call, it works much better with  
PETSC_COMM_SELF, because then I don't have any MPI calls.

What I was trying to do is basically determine the nonzero elements  
of a matrix row by row and then use this information to get exactly  
those entries from a different matrix. The actual code was too long  
and I just included everything that was needed for a crash.

Thanks for your help,
Arvid


>  Matt
>
> On 1/12/07, Arvid Bessen <bessen at cs.columbia.edu> wrote:
>>
>> Dear all,
>>
>> I have a problem with MPI in PETSc that is probably simple and you
>> will hopefully know the answer to. In my code I treat the local rows
>> of a MPIAIJ matrix one by one. If the number of local rows is equal
>> on all processors, everything works well. However, if the number of
>> rows is different, the program crashes. I have attached a simple
>> example program to demonstrate what I want to do.
>> Suppose process A has one more local row than process B. In the
>> debugger I found out that the crash occurs in "MPI_Allreduce".
>> Process A will call MPI_Allreduce one more time than process B,
>> because it processes an additional row. While process A is still
>> waiting for the result of the last MPI_Allreduce, process B will
>> proceed and call MPI_Allreduce in a different context. This is, when
>> the program crashes.
>>
>> What is the best way to make sure that all MPI calls are matched on
>> the different processes?
>>
>> Thanks,
>> Arvid
>>
>> static char help[] = "Example program with different row sizes\n\n";
>>
>> #include "petscmat.h"
>>
>> #undef __FUNCT__
>> #define __FUNCT__ "main"
>> int main(int Argc,char **Args)
>> {
>>    const PetscInt  size = 3;
>>    PetscErrorCode  ierr;
>>    Mat             cmat, dmat;
>>    PetscInt        cmat_loc_start, cmat_loc_end, cmat_loc_size;
>>    PetscInt        *rowsize_arr = 0;
>>    const PetscInt  **row_arr = 0;
>>    IS              *is_arr = 0;
>>    PetscInt        a;
>>    Mat            *submat_arr = 0;
>>
>>    PetscInitialize(&Argc,&Args,(char *)0,help);
>>
>>    ierr = MatCreateMPIAIJ(PETSC_COMM_WORLD, PETSC_DECIDE,
>> PETSC_DECIDE, size, size,
>>                          1, PETSC_NULL, 2, PETSC_NULL,  
>> &cmat);CHKERRQ(ierr);
>>    ierr = MatAssemblyBegin(cmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>    ierr = MatAssemblyEnd(cmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>    ierr = MatCreateMPIAIJ(PETSC_COMM_WORLD, PETSC_DECIDE,
>> PETSC_DECIDE, size, size,
>>                          1, PETSC_NULL, 2, PETSC_NULL,  
>> &dmat);CHKERRQ(ierr);
>>    ierr = MatAssemblyBegin(dmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>    ierr = MatAssemblyEnd(dmat,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>
>>    ierr = MatGetOwnershipRange(cmat, &cmat_loc_start, &cmat_loc_end);
>> CHKERRQ(ierr);
>>    cmat_loc_size = cmat_loc_end-cmat_loc_start;
>>
>>    ierr = PetscMalloc(sizeof(PetscInt *)*cmat_loc_size, &row_arr);
>> CHKERRQ(ierr);
>>    ierr = PetscMalloc(sizeof(PetscInt)*cmat_loc_size, &rowsize_arr);
>> CHKERRQ(ierr);
>>    ierr = PetscMalloc(sizeof(IS)*cmat_loc_size, &is_arr); CHKERRQ 
>> (ierr);
>>
>>    for( a=0; a<cmat_loc_size; a++ ) {
>>      ierr = MatGetRow(cmat, a+cmat_loc_start, &(rowsize_arr[a]), &
>> (row_arr[a]), 0); CHKERRQ(ierr);
>>      ierr = ISCreateGeneral(PETSC_COMM_WORLD, rowsize_arr[a], row_arr
>> [a], &(is_arr[a])); CHKERRQ(ierr);
>>      ierr = MatRestoreRow(cmat, a+cmat_loc_start, &(rowsize_arr 
>> [a]), &
>> (row_arr[a]), 0); CHKERRQ(ierr);
>>    }
>>
>>    ierr = PetscMalloc(sizeof(Mat)*cmat_loc_size, &submat_arr);  
>> CHKERRQ
>> (ierr);
>>    ierr = MatGetSubMatrices(dmat, cmat_loc_size, is_arr, is_arr,
>> MAT_INITIAL_MATRIX, &submat_arr); CHKERRQ(ierr);
>>
>>    ierr = PetscFinalize();CHKERRQ(ierr);
>>    return 0;
>> }
>>
>>
>>
>
>
> -- 
> One trouble is that despite this system, anyone who reads journals  
> widely
> and critically is forced to realize that there are scarcely any  
> bars to eventual
> publication. There seems to be no study too fragmented, no  
> hypothesis too
> trivial, no literature citation too biased or too egotistical, no  
> design too
> warped, no methodology too bungled, no presentation of results too
> inaccurate, too obscure, and too contradictory, no analysis too  
> self-serving,
> no argument too circular, no conclusions too trifling or too  
> unjustified, and
> no grammar and syntax too offensive for a paper to end up in print. --
> Drummond Rennie