[petsc-users] Help with DMDASetBlockFillsSparse

[Barry Smith]

  Alex,

 Ahh, some over-zealous error checking used for the sparse case that is not there otherwise. 

 You can edit src/dm/impls/da/fdda.c and remove the lines

  if (bx == DM_BOUNDARY_PERIODIC && (m % col)) SETERRQ(PetscObjectComm((PetscObject)da),PETSC_ERR_SUP,"For coloring efficiency ensure number of grid points in X is divisible\n\
                 by 2*stencil_width + 1\n");
  if (by == DM_BOUNDARY_PERIODIC && (n % col)) SETERRQ(PetscObjectComm((PetscObject)da),PETSC_ERR_SUP,"For coloring efficiency ensure number of grid points in Y is divisible\n\
                 by 2*stencil_width + 1\n");
  if (bz == DM_BOUNDARY_PERIODIC && (p % col)) SETERRQ(PetscObjectComm((PetscObject)da),PETSC_ERR_SUP,"For coloring efficiency ensure number of grid points in Z is divisible\n\
                 by 2*stencil_width + 1\n");

  then do make libs in a PETSc directory. 

  Regarding the slowdown. What becomes slower? Creating the DM or somewhere later in the code?

  My guess is that what becomes slow is your matrix assembly process and this happens because you provide the "wrong" sparsity pattern. That is your matrix actually has some entries that are not represented in the sparsity pattern. It will take forever to fill the matrix if this is the case. You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE) before putting values into the matrix to see if that is the case.

  Barry



> On Aug 13, 2020, at 9:16 PM, Alex Koshkarov <okoshkarov at tae.com> wrote:
> 
> Hello All,
>  
> I have a problem with DMDA matrix, I hope you can help. I am solving big PDE with TS object and my main data structure is 3D DMDA which has large number of degrees of freedom. I solve this system implicitly with JFNK. Essentially, I do something like this (J is jacobian, P is preconditioner):
>  
> MatCreateSNESMF(snes,&J);   
> DMCreateMatrix(da,&P);
> SNESSetJacobian(snes,J,P,form_P,env);
>  
> In form_P I form P, but not J, and all works. I also sometimes use 3D DMDA for 2D and 1D problems, just setting Ny=Nz=1 and periodic boundary conditions. I have many degrees of freedom in my DMDA, so I need blocks to be sparse. So I tried to use DMDASetBlockFillsSparse, before DMCreateMatrix. However, it gives me an error connected that size of each dimension should be divisible by (2*stencil_size +1) to make efficient coloring, which makes using 3D DMDA for 2D problems not optimal. Moreover, it becomes super slow… I am not sure why it needs coloring. Essentially, everything works fine without DMDASetBlockFillsSparse, but once I use it, it complains and becomes slow. Do I do something wrong here? Do I need this coloring? All I need is more sparse DMDA matrix… I am somewhat lost.
>  
> Thank you very much and best regards,
> Alex Koshkarov.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200813/a99c853a/attachment.html>