[petsc-dev] plans for preconditioners for SeqSELL

Tue Feb 13 05:40:27 CST 2018

Richard,

what are the performances you get with MATSELL in PFLOTRAN?

Today, I run the SELL matrix for the first time on a KNL with different
problems (using PetIGA)  and got the results attached.
It seems that SELL is faster then AIJ for 2D Poisson, and slightly faster
for 3D Poisson. However, for multi-component problems (i.e. Elasticity
and Cahn-Hilliard in mixed formulation) it has comparable performances or
slower.
in these cases AIJ is faster since it uses the inodes routines.

Is this expected?
These are the flags I used to compile PETSc on KNL --COPTFLAGS=-xMIC-AVX512
-O3 -mP2OPT_hpo_vec_remainder=F;

2018-02-12 21:04 GMT+03:00 Richard Tran Mills <rtmills at anl.gov>:

> On Mon, Feb 12, 2018 at 8:47 AM, Smith, Barry F. <bsmith at mcs.anl.gov>
> wrote:
>
>>
>>
>> > On Feb 12, 2018, at 10:25 AM, Stefano Zampini <
>> stefano.zampini at gmail.com> wrote:
>> >
>> > Barry,
>> >
>> > for sure Amat,Pmat is the right approach; however, with complicated
>> user codes, we are not always in control of having a different Jacobian
>> matrix.
>> > Since Mat*SELL does not currently support any preconditioning except
>> PCSOR and PCJACOBI, we ask the user to put codes like
>> >
>> > if (type is SELL)
>> >  create two matrices (and maybe modify the code in many other parts)
>> > else
>> >   ok with the previous code
>>
>>    I don't disagree with what you are saying and am not opposed to the
>> proposed work.
>>
>>    Perhaps we need to do a better job with making the mat,pmat approach
>> simpler or better documented so more people use it naturally in their
>> applications.
>>
>
> I wrote some code like that in some of the Jacobian/function routines in
> PFLOTRAN to experiment with MATSELL, and it works, but looks and feels
> pretty hacky. And if I wanted to support it for all of the different
> systems that PFLOTRAN can model, then I'd have to reproduce that it in many
> different Jacobian and function evaluation routines. I also don't like that
> it makes it awkward to play with the many combinations of matrix types and
> preconditioners that PETSc allows: The above pseudocode should really say
> "if (type is SELL) and (preconditioner is not PCSOR or PCJACOBI)". I do
> think that Amat,Pmat is a good approach in many situations, but it's easy
> to construct scenarios in which it falls short.
>
> In some situations, what I'd like to have happen is what Stefano is
> talking about, with an automatic conversion to AIJ happening if SELL
> doesn't support an operation. But, ideally, I think this sort of implicit
> format conversion shouldn't be something hard-coded into the workings of
> SELL. Instead, there should be some general mechanism by which PETSc
> recognizes that a particular operation is unsupported for a given matrix
> format, and then it can (optionally) copy/convert to a different matrix
> type (probably default to AIJ, but it shouldn't have to be AIJ) that
> supports the operation. This sort of implicit data rearrangement game may
> actually become more important if future computer architectures strongly
> prefer different data layouts different types of operations (though let's
> not get ahead of ourselves).
>
> --Richard
>
>
>>
>>     Barry
>>
>> >
>> > Just my two cents.
>> >
>> >
>> > 2018-02-12 19:10 GMT+03:00 Smith, Barry F. <bsmith at mcs.anl.gov>:
>> >
>> >
>> > > On Feb 12, 2018, at 9:59 AM, Stefano Zampini <
>> stefano.zampini at gmail.com> wrote:
>> > >
>> > > FYI, I just checked and MatSOR_*SELL does not use any vectorized
>> instruction.
>> > > Why just not converting to SeqAIJ, factor and then use the AIJ
>> implementation for MatSolve for the moment?
>> >
>> >   Why not use the mat, pmat feature of the solvers to pass in both
>> matrices and have the solvers handle using two formats simultaneously
>> instead of burdening the MatSELL code with tons of special code for
>> automatically converting to AIJ for solvers etc?
>> >
>> >
>> > >
>> > > 2018-02-12 18:06 GMT+03:00 Stefano Zampini <stefano.zampini at gmail.com
>> >:
>> > >
>> > >
>> > > 2018-02-12 17:36 GMT+03:00 Jed Brown <jed at jedbrown.org>:
>> > > Karl Rupp <rupp at iue.tuwien.ac.at> writes:
>> > >
>> > > > Hi Stefano,
>> > > >
>> > > >> Is there any plan to write code for native ILU/ICC etc for
>> SeqSELL, at least to have BJACOBI in parallel?
>> > > >
>> > > > (imho) ILU/ICC is a pain to do with SeqSELL. Point-Jacobi should be
>> > > > possible, yes. SELL is really just tailored to MatMults and a pain
>> for
>> > > > anything that is not very similar to a MatMult...
>> > >
>> > > There is already MatSOR_*SELL.  MatSolve_SeqSELL wouldn't be any
>> harder.
>> > > I think it would be acceptable to convert to SeqAIJ, factor, and
>> convert
>> > > the factors back to SELL.
>> > >
>> > > Yes, this was my idea. Today I have started coding something. I'll
>> push the branch whenever I have anything working
>> > >
>> > >
>> > >
>> > > --
>> > > Stefano
>> > >
>> > >
>> > >
>> > > --
>> > > Stefano
>> >
>> >
>> >
>> >
>> > --
>> > Stefano
>>
>>
>

-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20180213/16271c9c/attachment-0001.html>
-------------- next part --------------
#2D, Poisson, linear elements
$ ./Poisson -iga_view -iga_dim 2 -iga_elements {64|128|256|512} -iga_degree 1 -iga_continuity 0 -ksp_max_it 100 -pc_type none -log_view -iga_mat_type aij | grep -e "MatMult"
2.2647e-02
9.0335e-02
3.4515e-01
1.3588e+00

$ ./Poisson -iga_view -iga_dim 2 -iga_elements {64|128|256|512} -iga_degree 1 -iga_continuity 0 -ksp_max_it 100 -pc_type none -log_view -iga_mat_type sell | grep -e "MatMult"
7.3378e-03
3.1030e-02
1.1780e-01
4.7410e-01

#2D, Poisson, quadratic elements
$ ./Poisson -iga_view -iga_dim 2 -iga_elements {64|128|256|512} -iga_degree 2 -iga_continuity 0 -ksp_max_it 100 -pc_type none -log_view -iga_mat_type aij | grep -e "MatMult"
1.5822e-01
6.0223e-01
2.4154e+00
9.6371e+00

$ ./Poisson -iga_view -iga_dim 2 -iga_elements {64|128|256|512} -iga_degree 2 -iga_continuity 0 -ksp_max_it 100 -pc_type none -log_view -iga_mat_type sell | grep -e "MatMult"
6.4547e-02
2.5003e-01
9.5681e-01
3.7487e+00

#3D, Poisson, linear elements
$ ./Poisson -iga_view -iga_dim 3 -iga_elements {16|32|64|128} -iga_degree 1 -iga_continuity 0 -ksp_max_it 10 -pc_type none -log_view -iga_mat_type aij | grep -e "MatMult"
4.0069e-03
2.8143e-02
2.2254e-01
1.7520e+00

$ ./Poisson -iga_view -iga_dim 3 -iga_elements {16|32|64|128} -iga_degree 1 -iga_continuity 0 -ksp_max_it 10 -pc_type none -log_view -iga_mat_type sell | grep -e "MatMult"
2.3954e-03
1.6257e-02
1.2749e-01
1.1685e+00

#3D, Poisson, quadratic elements
$ ./Poisson -iga_view -iga_dim 3 -iga_elements {8|16|32|64} -iga_degree 2 -iga_continuity 0 -ksp_max_it 10 -pc_type none -log_view -iga_mat_type aij | grep -e "MatMult"
8.2695e-03
6.1627e-02
4.9080e-01
4.0070e+00

$ ./Poisson -iga_view -iga_dim 3 -iga_elements {8|16|32|64} -iga_degree 2 -iga_continuity 0 -ksp_max_it 10 -pc_type none -log_view -iga_mat_type sell | grep -e "MatMult"
8.3804e-03
5.7168e-02
4.5039e-01
3.9697e+00

# 2D, linear elasticity, linear elements
$ ./Elasticity -iga_view -iga_dim 2 -iga_elements {32|64|128|256} -iga_degree 1 -iga_continuity 0 -ksp_max_it 100 -pc_type none -log_view -iga_mat_type aij  | grep MatMult
1.4690e-02
6.0008e-02
2.1919e-01
9.0399e-01

$ ./Elasticity -iga_view -iga_dim 2 -iga_elements {32|64|128|256} -iga_degree 1 -iga_continuity 0 -ksp_max_it 100 -pc_type none -log_view -iga_mat_type sell  | grep MatMult
6.1235e-03
2.9640e-02
1.1271e-01
4.1062e-01

# 3D, linear elasticity, linear elements
$ ./Elasticity -iga_view -iga_dim 3 -iga_elements {8,16,32,64} -iga_degree 1 -iga_continuity 0 -ksp_max_it 100 -pc_type none -log_view -iga_mat_type aij  | grep MatMult
3.0665e-02
2.0822e-01
1.5074e+00
1.1917e+01

$ ./Elasticity -iga_view -iga_dim 3 -iga_elements {8,16,32,64} -iga_degree 1 -iga_continuity 0 -ksp_max_it 100 -pc_type none -log_view -iga_mat_type sell  | grep MatMult
3.6784e-02
2.2829e-01
1.7218e+00
1.3450e+01

# 2D, Cahn-Hilliard Mixed formulation, cubic elements, C^2 continuity
$ ./CahnHilliardMixed -dim 2 -deg 3 -nel {8|16|32} -cxx -iga_mat_type seqaij -ts_max_steps 1 -snes_error_if_not_converged 0 -ksp_max_it 1000 -pc_type none -ts_error_if_step_fails 0 -log_view  | grep MatMult
1.8025e-01
7.0823e-01
3.9769e+00

$ ./CahnHilliardMixed -dim 2 -deg 3 -nel {8|16|32} -cxx -iga_mat_type seqsell -ts_max_steps 1 -snes_error_if_not_converged 0 -ksp_max_it 1000 -pc_type none -ts_error_if_step_fails 0 -log_view  | grep MatMult
2.0002e-01
7.3748e-01
4.7165e+00