Question on using DMMG

Barry Smith bsmith at
Fri Feb 20 13:41:59 CST 2009

On Feb 20, 2009, at 12:34 PM, Randall Mackie wrote:

> Thanks Barry, this was useful and I now understand how to set the  
> levels and
> the grid size at the coarsest level.
> I'm trying to run a simple test problem, and it seems to be taking  
> *forever*
> in the routine setting the Jacobian. In my old code, where I use  
> to set up the matrix with the correct pre-allocation, it takes no  
> time to fill
> this matrix.

    First we need to see WHAT is taking so long (in my experience the  
only thing that takes "forever" is not have
enough preallocation; perhaps you do not have the correct DA stencil  
width or type or you are zeroing the matrix or
something before setting the values (thus throwing away the correct  

    Questions: 1) run with -info and send the result to petsc-maint at
2) what happens if you do not compute the Jacobian and just let PETSc  
do it.  That is run with -dmmg_jacobian_fd


> Here's a sketch of my ComputeJacobian routine:
> 	call DMMGGetDA
> 	Call DAGetGlobalIndices
> 	Loop through model:
> 	  v(ic)=fact
>          col(ic)=ltog(idltog +.....)
> 	  etc
> 	  call MatSetValues(...)
> I suspect this is a problem of the default stencil that is set up,  
> with
> many more entries than I'm using. Any way to just use  
> MATMPIAIJSetPreallocation,
> or do I need to use DASetBlockFills?
> Thanks, Randy
> Barry Smith wrote:
>> On Feb 19, 2009, at 3:48 PM, Randall Mackie wrote:
>>> Barry,
>>> I'm a little confused then, by how you set the size of the coarse  
>>> grid
>>> in relation to the fine grid. What if my fine grid is 283x107x113?  
>>> Can
>>> I just use a coarse grid of 3x3x3 and then the finer ones are  
>>> generated
>>> until it gets to my fine grid? I've been trying to follow ex22f.F,  
>>> where
>>> you set the coarse grid, and number of levels, but I can't figure  
>>> out
>>> the relation between the grid size on each level and how you specify
>>                                                                                               ^ 
>> ^^^^^^^^^^^^^^^
>>> the grid size at the finest level.
>>    We don't specify the grid size at the finest level; it just  
>> comes out from the coarse
>> grid size and the number of levels.
>>> In ex22f.F, when I run it, I see the first level is 27x27 (since the
>>> 3D grid is 3x3x3). Then at the next level it goes to 125x125 or  
>>> 5x5x5.
>>> Then the final level is 729x729, or 9x9x9.
>>   By default, with nonperiodic domains the grid spacing (distance  
>> between grid points)
>> is cut in half with each refinement. So, for one dimension this  
>> looks like
>>      *                                  
>> *                              *
>>      +                +              +            +               +
>>      x       x       x      x      x     x      x      x       x
>> that's 3 points, then 5, then 9.
>>   To get the 113 with three levels then make it 29 on the coarsest,  
>> then it becomes 57 then it
>> becomes 113. For four levels start with 15.
>>    You can refine by other factors like 3 using  
>> DASetRefinementFactor() call this after
>> DACreate but before DMMGSetDA() You can also control this via the  
>> options database
>> see the manual page.
>> Barry
>>> Thanks, Randy
>>> Barry Smith wrote:
>>>> On Feb 19, 2009, at 3:22 PM, Randall Mackie wrote:
>>>>> I'm trying to modify my code to use the DMMG routines to solve  
>>>>> my system
>>>>> by multi-grid, instead of the usual krylov solver. In my  
>>>>> previous code,
>>>>> I would set up a DA for my finite difference grid, set the matrix
>>>>> preallocation using MATCreateMPIAIJ, then fill the matrix, etc.
>>>>> In the code I've modified to use DMMG, here is my sequence of  
>>>>> routines
>>>>> so far:
>>>>>   call DMMGCreate(...)
>>>>>   call DACreate3D(....) ! to set the DA on the finest level
>>>>>   call DMMGSetDM(...)
>>>>>   call DMMGSETKSP(...)
>>>>> Questions:
>>>>> 1) I want to set the DA on the finest level and use Galerkin to  
>>>>> get
>>>>> the coarser matrices. So, is the DACreate above the way to do  
>>>>> this?
>>>>  The current design always has you setting the coarsest DA; DMMG  
>>>> generates all the finer ones
>>>> automatically. This is independent of using Galerkin coarse grids  
>>>> or not.
>>>> To have DMMG use Galerkin to generate the coarser matrices call  
>>>> DMMGSetUseGalerkinCoarse()
>>>> after you se the number of levels or use the option -dmmg_galerkin
>>>>> 2) How do I set the matrix preallocation here for use in the  
>>>>> DMMG routines?
>>>>> Would I use the same MATCreateMPIAIJ as before, and if so, where  
>>>>> do I
>>>>> put that? In the routine that fills the Jacobian, or in the main  
>>>>> program
>>>>> someplace?
>>>>   DMMG manages all this for you; it creates the empty matrices  
>>>> (based on the DA stencil) and gives them
>>>> the right nonzero structure.
>>>> See, for example, src/ksp/ksp/examples/tutorials/ex34.c
>>>>  Barry
>>>>> Thanks, Randy

More information about the petsc-users mailing list