[petsc-users] Problem in MPI communicator

[Barry Smith]

   This may not be the only problem but you can never ever ever change PETSC_COMM_WORLD after PetscInitialize(). You need to

  1) call MPI_Init() your self first then 

  2) do your renumbering using MPI_COMM_WORLD, not PETSC_COMM_WORLD 

  3) set PETSC_COMM_WORLD to your new comm 

  4) call PetscInitialize().

Let us know how it goes, if you still get failure send the entire .F file so we can run it to track down the issue.

   Barry

On Jun 9, 2014, at 10:56 PM, Michele Rosso <michele.rosso84 at gmail.com> wrote:

> Hi,
> 
> I am trying to re-number the mpi ranks in order to have the domain decomposition obtained from DMDACreate3D() match the default decomposition provided by MPI_Cart_create(). I followed the method described in the FAQ:
> 
>        call PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>        call mpi_comm_rank(MPI_COMM_WORLD, rank, ierr)
>        if(PETSC_COMM_WORLD/=MPI_COMM_WORLD) write(*,*) 'Communicator problem'
>        x = rank / (pz*py);
>        y = mod(rank,(pz*py))/pz
>        z = mod(mod(rank,pz*py),pz)
>        newrank = z*py*px + y*px + x;
>        call mpi_comm_split(PETSC_COMM_WORLD, 1, newrank, newcomm, ierr)
>        PETSC_COMM_WORLD = newcomm
> 
> I tried to run my code (it works fine with the standard PETSc decomposition) with the new decomposition but I received the error message;  I attached the full output. I run with only one processor to test the setup and I commented all the lines where I actually insert/get data into/from the PETSc arrays.
> Could you please help fixing this?
> 
> Thanks,
> Michele
> 
> 
> 
> 
> 
> <outfile.txt>