Cg/asm doesn't scale

[Matthew Knepley]

On Thu, Apr 23, 2009 at 2:13 PM, Nguyen, Hung V ERDC-ITL-MS <
Hung.V.Nguyen at usace.army.mil> wrote:

> Hello Matt,
>
> >ILU is incredibly unpredictable.
>
> I got the same result when running without setting ILU in sub_pc_type. It
> seems to me that direct solver is set up as default when solving for
> sub_ksp_type(?). Please let me know if it is not correct.
>

You can see what the default is using -ksp_view


> >You have not provided the Jacobi numbers, but the particular nonzero
> pattern
> of the non-overlapping matrix must be
> >much more amenable. Also, this is a really really bad preconditioner for
> your system.
>
> Indeed, the asm preconditioner is not really bad preconditioner for some of
> my ill-conditioned systems. In some other SPD linear systems, I have found
> that cg with asm preconditioner converges better than others. And, it does
> scale well within the size of matrix, see attached file. However, it
> doesn't
> scale in this case. Here is the solver time for cg/jacobi. The performance
> of
> cg/asm is better than cg/jacobi in the range from 1 to 4 processors.
>
>        Number Pes   Solver Time (secs)   #it      Solver Time(secs)
>                     cg/asm                          cg/jacobi
>        1                31.317345       544         1999.276566
>        2               263.172225       6959        1188.067975
>        4             734.828840        23233         984.062940
>        8             805.217591        41250         538.102407
>        16            611.813716        49262         308.547316
>        32            345.331928        49792         170.074248
>        64            212.084555        53771          92.398144
>
> >I would put my time into figuring out why my system is so ill-conditioned
> and try to formulate a good preconditioner, like an approximate system,
> etc.
>
> The linear system is from groundwater flow in a water repellent soil that
> can
> cause a very ill-conditioned linear system.


This is why people develop special purpose discretizations for these
problems.

  Matt


>
> -Hung
>
> -----Original Message-----
> From: petsc-users-bounces at mcs.anl.gov
> [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley
> Sent: Thursday, April 23, 2009 11:12 AM
> To: PETSc users list
> Subject: Re: Cg/asm doesn't scale
>
> On Thu, Apr 23, 2009 at 11:07 AM, Nguyen, Hung V ERDC-ITL-MS
> <Hung.V.Nguyen at usace.army.mil> wrote:
>
>
>        Hello,
>
>        I tried to solver the SPD linear system with using cg/asm
> preconditioner and
>        found that it doesn't scale well, see table below. Note: it does
> scale well
>        with cg/jacobi preconditioner.
>
>        Do you know why it doesn't scale?
>
>
> ILU is incredibly unpredictable. You have not provided the Jacobi numbers,
> but the particular nonzero pattern of the non-overlapping matrix must be
> much
> more amenable. Also, this is a really really bad preconditioner for your
> system. I would put my time into figuring out why my system is so
> ill-conditioned and try to formulate a good preconditioner, like an
> approximate system, etc.
>
>  Matt
>
>
>
>        Thanks,
>
>        -hung
>
>        Number Pes   Solver Time (secs)  #it
>        1                31.317345         544
>        2               263.172225        6959
>        4             734.828840        23233
>        8             805.217591        41250
>        16            611.813716        49262
>        32            345.331928        49792
>        64            212.084555        53771
>
>
>        ---
>        1 :  aprun -n 1  ./test_matrix_read  -ksp_type cg -pc_type asm
> -pc_asm_type
>        basic   -sub_pc_type ilu -sub_ksp_type preonly -ksp_rtol 1.0e-12
> -ksp_max_it
>        100000
>        Time in PETSc solver: 31.317345 seconds
>        The number of iteration       = 544
>        The solution residual error = 1.658653e-08
>        2 norm 7.885361e-07
>         infinity norm 6.738382e-09
>         1 norm 2.124207e-04
>
>        Application 679466 resources: utime 0, stime 0
>        ************************ Beginning new run ************************
>
>        2 :  aprun -n 2  ./test_matrix_read  -ksp_type cg -pc_type asm
> -pc_asm_type
>        basic   -sub_pc_type ilu -sub_ksp_type preonly -ksp_rtol 1.0e-12
> -ksp_max_it
>        100000
>        Time in PETSc solver: 263.172225 seconds
>        The number of iteration       = 6959
>        The solution residual error = 1.794494e-08
>        2 norm 6.579571e-07
>         infinity norm 8.745052e-09
>         1 norm 1.907733e-04
>
>        -- Here is info about matrix A:
>
>        Computed <structure:nrows> as <178353>
>        Computed <structure:symmetry> as <0>
>        Computed <structure:nnzeros> as <3578321>
>        Computed <structure:max-nnzeros-per-row> as <27>
>        Computed <structure:min-nnzeros-per-row> as <6>
>        Computed <structure:left-bandwidth> as <76553>
>        Computed <structure:right-bandwidth> as <76553>
>
>
>
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their experiments
is infinitely more interesting than any results to which their experiments
lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20090423/a5582b4c/attachment.htm>