Question on using DMMG

[Barry Smith]

On Feb 20, 2009, at 12:34 PM, Randall Mackie wrote:

> Thanks Barry, this was useful and I now understand how to set the  
> levels and
> the grid size at the coarsest level.
>
> I'm trying to run a simple test problem, and it seems to be taking  
> *forever*
> in the routine setting the Jacobian. In my old code, where I use  
> MATCreateMPIAIJ
> to set up the matrix with the correct pre-allocation, it takes no  
> time to fill
> this matrix.
>

    First we need to see WHAT is taking so long (in my experience the  
only thing that takes "forever" is not have
enough preallocation; perhaps you do not have the correct DA stencil  
width or type or you are zeroing the matrix or
something before setting the values (thus throwing away the correct  
preallocation).

    Questions: 1) run with -info and send the result to petsc-maint at mcs.anl.gov
2) what happens if you do not compute the Jacobian and just let PETSc  
do it.  That is run with -dmmg_jacobian_fd

    Barry


> Here's a sketch of my ComputeJacobian routine:
>
> 	call DMMGGetDA
> 	Call DAGetGlobalIndices
>
> 	Loop through model:
>
> 	  v(ic)=fact
>          col(ic)=ltog(idltog +.....)
>
> 	  etc
>
> 	  call MatSetValues(...)
>
>
>
> I suspect this is a problem of the default stencil that is set up,  
> with
> many more entries than I'm using. Any way to just use  
> MATMPIAIJSetPreallocation,
> or do I need to use DASetBlockFills?
>
>
> Thanks, Randy
>
>
> Barry Smith wrote:
>> On Feb 19, 2009, at 3:48 PM, Randall Mackie wrote:
>>> Barry,
>>>
>>> I'm a little confused then, by how you set the size of the coarse  
>>> grid
>>> in relation to the fine grid. What if my fine grid is 283x107x113?  
>>> Can
>>> I just use a coarse grid of 3x3x3 and then the finer ones are  
>>> generated
>>> until it gets to my fine grid? I've been trying to follow ex22f.F,  
>>> where
>>> you set the coarse grid, and number of levels, but I can't figure  
>>> out
>>> the relation between the grid size on each level and how you specify
>>                                                                                               ^ 
>> ^^^^^^^^^^^^^^^
>>>
>>> the grid size at the finest level.
>>    We don't specify the grid size at the finest level; it just  
>> comes out from the coarse
>> grid size and the number of levels.
>>>
>>>
>>> In ex22f.F, when I run it, I see the first level is 27x27 (since the
>>> 3D grid is 3x3x3). Then at the next level it goes to 125x125 or  
>>> 5x5x5.
>>> Then the final level is 729x729, or 9x9x9.
>>   By default, with nonperiodic domains the grid spacing (distance  
>> between grid points)
>> is cut in half with each refinement. So, for one dimension this  
>> looks like
>>      *                                  
>> *                              *
>>      +                +              +            +               +
>>      x       x       x      x      x     x      x      x       x
>> that's 3 points, then 5, then 9.
>>   To get the 113 with three levels then make it 29 on the coarsest,  
>> then it becomes 57 then it
>> becomes 113. For four levels start with 15.
>>    You can refine by other factors like 3 using  
>> DASetRefinementFactor() call this after
>> DACreate but before DMMGSetDA() You can also control this via the  
>> options database
>> see the manual page.
>> Barry
>>>
>>>
>>> Thanks, Randy
>>>
>>> Barry Smith wrote:
>>>> On Feb 19, 2009, at 3:22 PM, Randall Mackie wrote:
>>>>> I'm trying to modify my code to use the DMMG routines to solve  
>>>>> my system
>>>>> by multi-grid, instead of the usual krylov solver. In my  
>>>>> previous code,
>>>>> I would set up a DA for my finite difference grid, set the matrix
>>>>> preallocation using MATCreateMPIAIJ, then fill the matrix, etc.
>>>>>
>>>>> In the code I've modified to use DMMG, here is my sequence of  
>>>>> routines
>>>>> so far:
>>>>>
>>>>>   call DMMGCreate(...)
>>>>>   call DACreate3D(....) ! to set the DA on the finest level
>>>>>   call DMMGSetDM(...)
>>>>>
>>>>>
>>>>>   call DMMGSETKSP(...)
>>>>>
>>>>>
>>>>> Questions:
>>>>>
>>>>> 1) I want to set the DA on the finest level and use Galerkin to  
>>>>> get
>>>>> the coarser matrices. So, is the DACreate above the way to do  
>>>>> this?
>>>>  The current design always has you setting the coarsest DA; DMMG  
>>>> generates all the finer ones
>>>> automatically. This is independent of using Galerkin coarse grids  
>>>> or not.
>>>> To have DMMG use Galerkin to generate the coarser matrices call  
>>>> DMMGSetUseGalerkinCoarse()
>>>> after you se the number of levels or use the option -dmmg_galerkin
>>>>>
>>>>>
>>>>> 2) How do I set the matrix preallocation here for use in the  
>>>>> DMMG routines?
>>>>> Would I use the same MATCreateMPIAIJ as before, and if so, where  
>>>>> do I
>>>>> put that? In the routine that fills the Jacobian, or in the main  
>>>>> program
>>>>> someplace?
>>>>   DMMG manages all this for you; it creates the empty matrices  
>>>> (based on the DA stencil) and gives them
>>>> the right nonzero structure.
>>>> See, for example, src/ksp/ksp/examples/tutorials/ex34.c
>>>>  Barry
>>>>>
>>>>>
>>>>>
>>>>> Thanks, Randy