[petsc-users] Questions about PCMG
Yuqi Wu
ywu at culink.Colorado.EDU
Thu Apr 5 22:09:14 CDT 2012
Thank you, Barry.
Now I put in the KSPSetOperators() in the ComputeJacobian() routine. It works. And I found out some interesting things, it also triggers the LU factorization for the fine grid smoothers. Now I get 2 MatLUFactorSym (one for coarse and one the fine grid), and 4 MatLUFactorNum (two for coarse and two for fine) for 2 SNES iteration.
Can I also ask another question? Because MatPtAP is not supported for baij matrices. I want to setup a routine to calculate coarse operator if the fine grid matrices is of type baij. Is there any approach so that I can keep the symbolic pattern and only redo the numeric part in each SNES, similar to MatPtAPSymbolic and MatPtAPNumeric?
Best
Yuqi
---- Original message ----
>Date: Thu, 5 Apr 2012 21:35:58 -0500
>From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>Subject: Re: [petsc-users] Questions about PCMG
>To: PETSc users list <petsc-users at mcs.anl.gov>
>
>
>On Apr 5, 2012, at 9:31 PM, Yuqi Wu wrote:
>
>> Hi, Barry,
>>
>> If I use the following routines to setup the coarse operator instead of calling PCMGSetGalerkin()
>>
>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>
> Are you calling this at every SNES iteration? You should call this inside your ComputeJacobian() function, is that where you call it? Passing the DIFFERENT_NONZERO_PATTERN will also trigger a new symbolic factorization also.
>
> The refactorization fo the find grid smoother not being redone only effects the fine grid smoother, not the coarse grid.
> Barry
>
>
>>
>> Then in the case I have 2 SNES iteration, the MatLUFactorSym and MatLUFactorNum are only called twice in the program. Is this issue also related to the refactorization issue for the fine grid smoother?
>>
>> Best
>>
>> Yuqi
>>
>>
>>
>> ---- Original message ----
>>> Date: Thu, 5 Apr 2012 14:10:02 -0500
>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>> Subject: Re: [petsc-users] Questions about PCMG
>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>
>>>
>>> Yes, at least that is the intention.
>>>
>>> Barry
>>>
>>> On Apr 5, 2012, at 2:06 PM, Yuqi Wu wrote:
>>>
>>>> Hi, Barry,
>>>>
>>>> If my matrix is of type mpibaij, then I cannot use the Galerkin flag in the PCMG, e.g. PCMGSetGalerkin(finepc,PETSC_TRUE);
>>>>
>>>> Otherwise, I got the following error messages in PETSc
>>>> [0]PETSC ERROR: No support for this operation for this object type!
>>>> [0]PETSC ERROR: Matrix of type <mpibaij> does not support PtAP!
>>>>
>>>> Can I use the following commands to setup the coarse operator
>>>>
>>>> ierr = PCMGGetCoarseSolve(finepc,&ctx->coarseksp);CHKERRQ(ierr);
>>>> ierr = KSPSetOperators(ctx->coarseksp, coarsealgebra->J, coarsealgebra->J, DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
>>>>
>>>> Thank you.
>>>>
>>>> Best
>>>>
>>>> Yuqi
>>>>
>>>>
>>>> ---- Original message ----
>>>>> Date: Thu, 5 Apr 2012 08:10:03 -0500
>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>
>>>>>
>>>>> It looks like the logic for that line has gotten so convoluted that it lost its purpose and is not triggering the rebuilding on the finest level. Thanks for finding this, we will investigate and get back to you.
>>>>>
>>>>>
>>>>> Barry
>>>>>
>>>>> If your code is not complicated and you are willing to send it to us at petsc-maint at mcs.anl.gov that will make it quicker for us to determine the problem.
>>>>>
>>>>>
>>>>> On Apr 4, 2012, at 11:55 PM, Yuqi Wu wrote:
>>>>>
>>>>>> Thank you Barry.
>>>>>>
>>>>>> I am using exact LU as subdomain solver, so I find out PCSetUp_LU() in my debugging output. But there are only three PCSetUp_LU() was called in my program. One in the down smoother of the first SNES, one in the coarse solve of the first SNES, and one in the coarse solve of the second SNES.
>>>>>>
>>>>>> In those PCSetUp_LU(), MatLUFactorSymbolic and MatLUFactorNumeric is called. But I still have no ideas about the actions taken on the down smoother of the second SNES.
>>>>>>
>>>>>> I try to setup a breakpoint for PCSetUp_MG(). I find out that the IF statement shown in below is called in the first SNES but not the second SNES. Is it has something related to this issue?
>>>>>>
>>>>>> /* If user did not provide fine grid operators OR operator was not updated since last global KSPSetOperators() */
>>>>>> /* so use those from global PC */
>>>>>> /* Is this what we always want? What if user wants to keep old one? */
>>>>>> ierr = KSPGetOperatorsSet(mglevels[n-1]->smoothd,PETSC_NULL,&opsset);CHKERRQ(ierr);
>>>>>> ierr = KSPGetPC(mglevels[0]->smoothd,&cpc);CHKERRQ(ierr);
>>>>>> ierr = KSPGetPC(mglevels[n-1]->smoothd,&mpc);CHKERRQ(ierr);
>>>>>> if (!opsset || ((cpc->setupcalled == 1) && (mpc->setupcalled == 2)) || ((mpc == cpc) && (mpc->setupcalled == 2))) {
>>>>>> ierr = PetscInfo(pc,"Using outer operators to define finest grid operator \n because PCMGGetSmoother(pc,nlevels-1,&ksp);KSPSetOperators(ksp,...); was not called.\n");CHKERRQ(ierr);
>>>>>> ierr = KSPSetOperators(mglevels[n-1]->smoothd,pc->mat,pc->pmat,pc->flag);CHKERRQ(ierr);
>>>>>> }
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>> Yuqi
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---- Original message ----
>>>>>>> Date: Wed, 4 Apr 2012 19:51:23 -0500
>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>
>>>>>>>
>>>>>>> Step through the PCSetUp() function using "next" and you will it first making the decision of whether to continue with the setup or simply return early.
>>>>>>>
>>>>>>> When it gets to the lines
>>>>>>>
>>>>>>> if (pc->ops->setup) {
>>>>>>> ierr = (*pc->ops->setup)(pc);CHKERRQ(ierr);
>>>>>>> }
>>>>>>>
>>>>>>> don't use "next" instead use "step" and it will go into the function PCSetUp_ILU() and you can move through that one with "next"
>>>>>>>
>>>>>>>> I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger.
>>>>>>>
>>>>>>> It has to be calling them. Did you use "break MatLUFactorSymbolic" in the debugger?
>>>>>>>
>>>>>>> Barry
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Apr 4, 2012, at 7:45 PM, Yuqi Wu wrote:
>>>>>>>
>>>>>>>> ---- Original message ----
>>>>>>>>> Date: Wed, 4 Apr 2012 13:55:55 -0500
>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Note: In most applications the flag SAME_NONZERO_PATTERN is provided in the compute Jacobian routine, this means that the SYMBOLIC factorization needs to be only done ONCE per matrix; only the numeric factorization needs to be done when the nonzero values have changed (the symbolic need not be repeated). Are you using this flag? How many times in the NUMERIC factorization being done?
>>>>>>>>>
>>>>>>>>> When you run the program with -info it will print information of the form: (run on one process to make life simple)
>>>>>>>>>
>>>>>>>>> Setting PC with identical preconditioner\
>>>>>>>>> Setting up new PC
>>>>>>>>> Setting up PC with same nonzero pattern\
>>>>>>>>> Setting up PC with different nonzero pattern\n
>>>>>>>>>
>>>>>>>>> How many, and exactly what messages of this form are you getting?
>>>>>>>>>
>>>>>>>>> When all else fails you can run the program in the debugger to track what is happening and why.
>>>>>>>>>
>>>>>>>>> Put a breakpoint in PCSetUp() then each time it gets called use next to step through it to see what is happening.
>>>>>>>>>
>>>>>>>>> First thing to check, is PCSetUp() getting called on each level for each new SNES iteration?
>>>>>>>>>
>>>>>>>>> Second thing, if it is then why is it not triggering the new numerical factorization.
>>>>>>>>
>>>>>>>> I run the program in debugger. I got total 8 PCSetUp() in the program. And I believe that PCSetup() is called on each level for each new SNES iteration.
>>>>>>>>
>>>>>>>> But I don't know how to check why is it not triggering the new numerical factorization in the debugger. I cannot find any MatLUFactorSymbolic or MatLUFactorNumeric calls in the debugger. Could you give me some suggestions to find out the problem?
>>>>>>>>
>>>>>>>> Best
>>>>>>>>
>>>>>>>> Yuqi
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Barry
>>>>>>>>>
>>>>>>>>> On Apr 4, 2012, at 1:34 PM, Yuqi Wu wrote:
>>>>>>>>>
>>>>>>>>>> Thanks, Adam.
>>>>>>>>>>
>>>>>>>>>> Yes. I am using the Galerkin coarse grids. But I am not sure whether the coarse grid is not getting refactored in the second SNES solve or the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>>>>
>>>>>>>>>> In the -info output attached in the previous email, the fine grid matrix is of size 11585 by 11585, and the coarse grid matrix is of size 4186 by 4186. In the -info output, I found out three MatLUFactorSymbolic_SeqAIJ routines, one for fine martrix, and two for coarse matrix.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 2 Fill ratio:given 5 needed 11.401
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 11.401 or use
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,11.401);
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 8057 nodes of 11585. Limit used: 5. Using Inode routines
>>>>>>>>>>
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>>>>
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Reallocs 1 Fill ratio:given 5 needed 7.07175
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): Run with -pc_factor_fill 7.07175 or use
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): PCFactorSetFill(pc,7.07175);
>>>>>>>>>> [0] MatLUFactorSymbolic_SeqAIJ(): for best performance.
>>>>>>>>>> [0] Mat_CheckInode_FactorLU(): Found 1764 nodes of 4186. Limit used: 5. Using Inode routines
>>>>>>>>>>
>>>>>>>>>> So I believe that the fine grid smoother is not getting refactored in the second SNES solve.
>>>>>>>>>>
>>>>>>>>>> Best
>>>>>>>>>>
>>>>>>>>>> Yuqi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---- Original message ----
>>>>>>>>>>> Date: Wed, 4 Apr 2012 14:24:28 -0400
>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of "Mark F. Adams" <mark.adams at columbia.edu>)
>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>
>>>>>>>>>>> I would expect 4 calls to MatLUFactorSym here. It looks like the coarse grid is not getting refactored in the second SNES solve.
>>>>>>>>>>>
>>>>>>>>>>> Are you using Galerkin coarse grids? Perhaps you are not setting a new coarse grid with KSPSetOperator and so MG does not bother refactoring it.
>>>>>>>>>>>
>>>>>>>>>>> Mark
>>>>>>>>>>>
>>>>>>>>>>> On Apr 4, 2012, at 1:53 PM, Yuqi Wu wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>
>>>>>>>>>>>> Can I ask another question?
>>>>>>>>>>>>
>>>>>>>>>>>> In my log summary output, it shows that although there are two SNES iteration and total 9 linear iterations. The functions MatLUFactorSym and MatLUFactorNum are only called for three times.
>>>>>>>>>>>>
>>>>>>>>>>>> MatLUFactorSym 3 1.0 1.4073e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 1 0 0 0 2 1 0 0 0 2 0
>>>>>>>>>>>> MatLUFactorNum 3 1.0 3.2754e+01 1.0 9.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 31 97 0 0 0 32 97 0 0 0 280
>>>>>>>>>>>>
>>>>>>>>>>>> I checked the -info output. It shows that One MatLUFactorSymbolic_SeqAIJ() is called in down smoother of the first SNES, one MatLUFactorSymbolic_SeqAIJ() is called in the coarse solve of the first SNES, and one MatLUFactorSymbolic_SeqAIJ() is called in the down smoother of the second SNES.
>>>>>>>>>>>>
>>>>>>>>>>>> Do you have any ideas why there are 9 multigrid iterations, but only 3 MatLUFactorSymbolic calls in the program?
>>>>>>>>>>>>
>>>>>>>>>>>> Best
>>>>>>>>>>>>
>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 20:08:27 -0500
>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> There are two linear solves (for 1 SNES and 2 SNES) so there are two MGSetUp on each level. Then a total of 9 multigrid iterations (in both linear solves together) hence 9 smoother on level 0 (level 0 means coarse grid solve). One smooth down and one smooth up on level 1 hence 18 total smooths on level 1. 9 computation of residual on level 1 and 18 MgInterp because that logs both the restriction to level 0 and the interpolation back to level 1 and 18 = 9 + 9.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Apr 3, 2012, at 7:57 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi, Barry,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you. If my program converges in two SNES iteration,
>>>>>>>>>>>>>> 0 SNES norm 1.014991e+02, 0 KSP its (nan coarse its average), last norm 0.000000e+00
>>>>>>>>>>>>>> 1 SNES norm 9.925218e-05, 4 KSP its (5.25 coarse its average), last norm 2.268574e-06.
>>>>>>>>>>>>>> 2 SNES norm 1.397282e-09, 5 KSP its (5.20 coarse its average), last norm 1.312605e-12.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And -pc_mg_log shows the following output
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> MGSetup Level 0 2 1.0 3.4091e-01 2.1 0.00e+00 0.0 3.0e+02 6.0e+04 3.0e+01 1 0 3 11 2 1 0 3 11 2 0
>>>>>>>>>>>>>> MGSmooth Level 0 9 1.0 1.2126e+01 1.0 9.38e+08 3.2 2.8e+03 1.7e+03 6.4e+02 33 71 28 3 34 35 71 28 3 35 415
>>>>>>>>>>>>>> MGSetup Level 1 2 1.0 1.3925e-01 2.1 0.00e+00 0.0 1.5e+02 3.1e+04 2.3e+01 0 0 1 3 1 0 0 1 3 1 0
>>>>>>>>>>>>>> MGSmooth Level 1 18 1.0 5.8493e+00 1.0 3.66e+08 3.1 1.5e+03 2.9e+03 3.6e+02 16 28 15 3 19 17 28 15 3 19 339
>>>>>>>>>>>>>> MGResid Level 1 9 1.0 1.1826e-01 1.4 1.49e+06 2.4 2.0e+02 2.7e+03 9.0e+00 0 0 2 0 0 0 0 2 0 0 70
>>>>>>>>>>>>>> MGInterp Level 1 18 1.0 1.2317e-01 1.3 7.74e+05 2.2 3.8e+02 1.1e+03 1.8e+01 0 0 4 0 1 0 0 4 0 1 37
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What are the MGSmooth, MGResid, MGInterp represent for?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yuqi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---- Original message ----
>>>>>>>>>>>>>>> Date: Tue, 3 Apr 2012 19:19:23 -0500
>>>>>>>>>>>>>>> From: petsc-users-bounces at mcs.anl.gov (on behalf of Barry Smith <bsmith at mcs.anl.gov>)
>>>>>>>>>>>>>>> Subject: Re: [petsc-users] Questions about PCMG
>>>>>>>>>>>>>>> To: PETSc users list <petsc-users at mcs.anl.gov>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -pc_mg_log doesn't have anything to do with DA or DMMG it is part of the basic PCMG. Are you sure you are calling SNESSetFromOptions()?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Apr 3, 2012, at 6:56 PM, Yuqi Wu wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi, Mark,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you so much for your suggestion.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The problem 1 is resolved by avoiding calling PCMGSetNumberSmoothUp.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But since I am using the unstructured grid in my application, I didn't use DA or dmmg, so -pc_mg_log didn't give any level information. I try to run my code using -info with 1 processor, and I find out some interesting issues.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>
More information about the petsc-users
mailing list