On Fri, Feb 20, 2009 at 12:34 PM, Randall Mackie <span dir="ltr"><<a href="mailto:rlmackie862@gmail.com">rlmackie862@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Thanks Barry, this was useful and I now understand how to set the levels and<br>
the grid size at the coarsest level.<br>
<br>
I'm trying to run a simple test problem, and it seems to be taking *forever*<br>
in the routine setting the Jacobian. In my old code, where I use MATCreateMPIAIJ<br>
to set up the matrix with the correct pre-allocation, it takes no time to fill<br>
this matrix.<br>
<br>
Here's a sketch of my ComputeJacobian routine:<br>
<br>
call DMMGGetDA<br>
Call DAGetGlobalIndices<br>
<br>
Loop through model:<br>
<br>
v(ic)=fact<br>
col(ic)=ltog(idltog +.....)<br>
<br>
etc<br>
<br>
call MatSetValues(...)<br>
<br>
<br>
<br>
I suspect this is a problem of the default stencil that is set up, with<br>
many more entries than I'm using. Any way to just use MATMPIAIJSetPreallocation,<br>
or do I need to use DASetBlockFills?</blockquote><div><br>1) More entires would not cause a slowdown, but fewer would.<br><br>2) I believe that DA matrices are flagged to generate an error when a malloc is<br> generated. You can run with -log to check this.<br>
<br> Matt<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
Thanks, Randy<br>
<br>
<br>
Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
On Feb 19, 2009, at 3:48 PM, Randall Mackie wrote:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Barry,<br>
<br>
I'm a little confused then, by how you set the size of the coarse grid<br>
in relation to the fine grid. What if my fine grid is 283x107x113? Can<br>
I just use a coarse grid of 3x3x3 and then the finer ones are generated<br>
until it gets to my fine grid? I've been trying to follow ex22f.F, where<br>
you set the coarse grid, and number of levels, but I can't figure out<br>
the relation between the grid size on each level and how you specify<br>
</blockquote>
^^^^^^^^^^^^^^^^<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
the grid size at the finest level.<br>
</blockquote>
<br>
We don't specify the grid size at the finest level; it just comes out from the coarse<br>
grid size and the number of levels.<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
In ex22f.F, when I run it, I see the first level is 27x27 (since the<br>
3D grid is 3x3x3). Then at the next level it goes to 125x125 or 5x5x5.<br>
Then the final level is 729x729, or 9x9x9.<br>
</blockquote>
<br>
By default, with nonperiodic domains the grid spacing (distance between grid points)<br>
is cut in half with each refinement. So, for one dimension this looks like<br>
<br>
* * *<br>
+ + + + +<br>
x x x x x x x x x<br>
<br>
that's 3 points, then 5, then 9.<br>
<br>
To get the 113 with three levels then make it 29 on the coarsest, then it becomes 57 then it<br>
becomes 113. For four levels start with 15.<br>
<br>
You can refine by other factors like 3 using DASetRefinementFactor() call this after<br>
DACreate but before DMMGSetDA() You can also control this via the options database<br>
see the manual page.<br>
<br>
Barry<br>
<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
Thanks, Randy<br>
<br>
Barry Smith wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
On Feb 19, 2009, at 3:22 PM, Randall Mackie wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I'm trying to modify my code to use the DMMG routines to solve my system<br>
by multi-grid, instead of the usual krylov solver. In my previous code,<br>
I would set up a DA for my finite difference grid, set the matrix<br>
preallocation using MATCreateMPIAIJ, then fill the matrix, etc.<br>
<br>
In the code I've modified to use DMMG, here is my sequence of routines<br>
so far:<br>
<br>
call DMMGCreate(...)<br>
call DACreate3D(....) ! to set the DA on the finest level<br>
call DMMGSetDM(...)<br>
<br>
<br>
call DMMGSETKSP(...)<br>
<br>
<br>
Questions:<br>
<br>
1) I want to set the DA on the finest level and use Galerkin to get<br>
the coarser matrices. So, is the DACreate above the way to do this?<br>
</blockquote>
The current design always has you setting the coarsest DA; DMMG generates all the finer ones<br>
automatically. This is independent of using Galerkin coarse grids or not.<br>
To have DMMG use Galerkin to generate the coarser matrices call DMMGSetUseGalerkinCoarse()<br>
after you se the number of levels or use the option -dmmg_galerkin<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
2) How do I set the matrix preallocation here for use in the DMMG routines?<br>
Would I use the same MATCreateMPIAIJ as before, and if so, where do I<br>
put that? In the routine that fills the Jacobian, or in the main program<br>
someplace?<br>
</blockquote>
DMMG manages all this for you; it creates the empty matrices (based on the DA stencil) and gives them<br>
the right nonzero structure.<br>
See, for example, src/ksp/ksp/examples/tutorials/ex34.c<br>
Barry<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
<br>
Thanks, Randy<br>
</blockquote></blockquote></blockquote>
<br>
</blockquote>
</blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<br>