[petsc-users] Argument out of range error only in certain mpi sizes

[Smith, Barry F.]

   Sajid,

     This code won't work. You are assuming that neighbors to the north and south can be obtained from set_col = i + My and set_col = i - My. This is simply incorrect. The global number of unknowns in PETSc is by process (not natural ordering on the entire domain) this is why the PETSc examples that use DMDA us either MatSetValueStencil() or MatSetValuesLocal(). Since your problem has a simple stencil you should use the MatSetValuesStencil() approach and loop over the local part of the grid, not over the local rows of the matrix. Copy one of the src/ksp/ksp/examples/tutorials examples as a starting point.

   Barry

A side note not directly related to the current problem.  You are storing both physical quantities (lambs, energy) and grid quantities (mx, my ...) in the AppCtx. We don't recommend this, better to just store the physical quantities there and to retrieve the grid quantities in the code from the DMDA object whenever you need them (usually with calls to DMDAGetInfo() instead of hardwiring them into the AppCtx. The reason you want to separate the problem specific stuff from the grid specific stuff is so that your functions can be called for different DMDA (for examples calling RHSMatrixFreeSpace for compute the matrix for each level of multigrid) without having to change stuff in the AppCtx between each call to match the "current" grid.  Your current handling is not "wrong" it is just perhaps not the most desirable design.



> On Apr 10, 2019, at 12:58 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi PETSc developers, 
> 
> I wanted to convert my code that in which I was using general Vec/Mat to DMDA based grid management (nothing fancy just a 5-point complex stencil). For this, I created a DA object and created the global solution vector using this. This worked fine. 
> 
> Now, I created the matrix using DMCreateMatrix and fill it using the regular function. I get no error when I run the problem using mpirun -np 1 and I thought my matrix filling logic aligns with the DM non-zero locations (Star Stencil, width 1). With mpirun -np 2, no errors either. But with mpirun -np 4 or 8, I get errors which say : "Argument out of range/ Inserting a new nonzero at global row/column ... into matrix". 
> 
> I would switch over the logic provided in KSP/ex46.c for filling the Matrix via the MatSetStencil logic but I wanted to know what I was doing wrong in my current code since I've never seen an error that depends on number of mpi processes.
> 
> I'm attaching the code ( which works if the matrix is created without using the DA, i.e. comment out line 159, uncomment 161/162 and I'm doing this on a small grid to catch errors.). 
> 
> Thanks in advance for the help. 
> 
> 
> -- 
> Sajid Ali
> Applied Physics
> Northwestern University
> <ex_dmda.c>