[petsc-dev] no petsc on Edison

[Hong]

Mark,
I implemented scalable MatPtAP and did comparisons of three implementations
using ex56.c on alcf cetus machine (this machine has small memory,
1GB/core):
- nonscalable PtAP: use an array of length PN to do dense axpy
- scalable PtAP:       do sparse axpy without use of PN array
- hypre PtAP.

The results are attached. Summary:
- nonscalable PtAP is 2x faster than scalable, 8x faster than hypre PtAP
- scalable PtAP is 4x faster than hypre PtAP
- hypre uses less memory (see job.ne399.n63.np1000.sh)

Based on above observation, I set the default PtAP algorithm as
'nonscalable'.
When PN > local estimated nonzero of C=PtAP, then switch default to
'scalable'.
User can overwrite default.

For the case of np=8000, ne=599 (see job.ne599.n500.np8000.sh), I get
MatPtAP                   3.6224e+01 (nonscalable for small mats, scalable
for larger ones)
scalable MatPtAP     4.6129e+01
hypre                        1.9389e+02

I'm merging this work to next, then to master soon.

Hong

On Wed, Feb 8, 2017 at 7:25 PM, Mark Adams <mfadams at lbl.gov> wrote:

> Thanks Hong, that sounds great.
>
> I am weary of silent optimizations like you suggest but 2x is big! and
> failure is very bad. So I would vote for your suggestion.
>
> ex56 is an elasticity problem. It would be nice, now that you have this
> experimental setup in place, to compare with hypre on a 3D scalar
> problem. Hypre might not spend much effort optimizing for block matrices.
> 3x better than hypre seems large to me.  I have to suspect that hypre does
> not exploit blocked matrices as much as we do.
>
> Thanks again,
>
>
>
> On Wed, Feb 8, 2017 at 12:07 PM, Hong <hzhang at mcs.anl.gov> wrote:
>
>> I conducted tests on MatMatMult() and MatPtAP()
>> using petsc/src/ksp/ksp/examples/tutorials/ex56.c (gamg) on a 8-core
>> machine (petsc machine). The output file is attached.
>>
>> Summary:
>> 1) non-scalable MatMatMult() for mpiaij format is 2x faster than scalable
>> version. The major difference between the two is dense-axpy vs. sparse-axpy.
>>
>> Currently, we set non-scalable as default, which leads to problem when
>> running large problems.
>> How about setting default as
>>  - non-scalable for small to medium size matrices
>>  - scalable for larger ones, e.g.
>>
>> +    ierr = PetscOptionsEList("-matmatmult_via","Algorithmic
>> approach","MatMatMult",algTypes,nalg,algTypes[1],&alg,&flg);
>>
>> +    if (!flg) { /* set default algorithm based on B->cmap->N */
>> +      PetscMPIInt size;
>> +      ierr = MPI_Comm_size(comm,&size);CHKERRQ(ierr);
>> +      if ((PetscReal)(B->cmap->N)/size > 100000.0) alg = 0; /* scalable
>> algorithm */
>> +    }
>>
>> i.e., if user does NOT pick an algorithm, when ave cols per process >
>> 100k, use scalable implementation; otherwise, non-scalable version.
>>
>> 2) We do NOT have scalable implementation for MatPtAP() yet.
>> We have non-scalable PtAP and interface to Hypre's PtAP. Comparing the
>> two,
>> Petsc MatPtAP() is approx 3x faster than Hypre's.
>>
>> I'm writing a scalable MatPtAP() now.
>>
>> Hong
>>
>> On Thu, Feb 2, 2017 at 2:54 PM, Stefano Zampini <
>> stefano.zampini at gmail.com> wrote:
>>
>>>
>>>
>>> Il 02 Feb 2017 23:43, "Mark Adams" <mfadams at lbl.gov> ha scritto:
>>>
>>>
>>>
>>> On Thu, Feb 2, 2017 at 12:02 PM, Stefano Zampini <
>>> stefano.zampini at gmail.com> wrote:
>>>
>>>> Mark,
>>>>
>>>> I saw your configuration has hypre. If you could run with master, you
>>>> may try -matptap_via hypre.
>>>>
>>>
>>> This is worth trying. Does this even work with GAMG?
>>>
>>>
>>> Yes, it should work, except that the block sizes, if any, are not
>>> propagated to the resulting matrix. I can add it if you need it.
>>>
>>>
>>>
>>> Treb: try hypre anyway. It has its own RAP code.
>>>
>>>
>>>
>>> With that option, you will use hypre's RAP with MATAIJ
>>>
>>>
>>> It uses BoomerAMGBuildCoarseOperator directly with the AIJ matrices.
>>>>
>>>> Stefano
>>>>
>>>> On Feb 2, 2017, at 7:28 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>>>
>>>>
>>>>
>>>> On Thu, Feb 2, 2017 at 11:13 AM, Hong <hzhang at mcs.anl.gov> wrote:
>>>>
>>>>> Mark:
>>>>> Try '-matmatmult_via scalable' first. If this works, should we set it
>>>>> as default?
>>>>>
>>>>
>>>> If it is robust I would say yes unless it is noticeably slower (say
>>>> >20%) small scale problems.
>>>>
>>>>
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20170227/92431069/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: out_ex56_cetus_short
Type: application/octet-stream
Size: 5377 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20170227/92431069/attachment.obj>