[petsc-users] MatCreate performance

[Matthew Knepley]

On Mon, Mar 11, 2019 at 8:23 AM Ale Foggia via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello all,
>
> Thanks for your answers.
>
> 1) I'm working with a matrix with a linear size of 2**34, but it's a
> sparse matrix, and the number of elements different from zero is
> 43,207,072,74. I know that the distribution of these elements is not
> balanced between the processes, the matrix is more populated in the middle
> part.
>
> 2) I initialize Slepc. Then I create the basis elements of the system
> (this part does not involve Petsc/Slepc, and every process is just
> computing -and owns- an equal amount of basis elements). Then I call:
> ierr = MatCreate(PETSC_COMM_WORLD, &A); CHKERRQ(ierr);
> ierr = MatSetType(A, MATMPIAIJ); CHKERRQ(ierr);
> ierr = MatSetSizes(A, PETSC_DECIDE, PETSC_DECIDE, size, size);
> CHKERRQ(ierr);
> ierr = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); CHKERRQ(ierr);
> ierr = MatZeroEntries(A); CHKERRQ(ierr);
> After this, I compute the elements of the matrix and set the values with
> MatSetValues. The I call EPSSolve (with KrylovSchur and setting the type as
> EPS_HEP).
>
> 3) There are a few more things that are strange to me. I measure the
> execution time of these parts both with a PetscLogStage and with a
> std::chrono (in nanoseconds) clock. I understand that the time given by the
> Log is an average over the processes right? In the case of the std::chrono,
> I'm only printing the times from process 0 (no average over processes).
> What I see is the following:
>                              1024 procs          2048 procs       4096
> procs        8192 procs
>                              Log       std          Log       std
> Log       std        Log       std
> MatCreate            68.42   122.7      67.08    121.2   62.29    116
>    73.36    127.4
> preallocation        140.36  140.3     76.45     76.45   40.31
> 40.3      21.13     21.12
> MatSetValues       237.79  237.7     116.6    116.6    60.59    60.59
> 35.32    35.32
> ESPSolve             162.8    160        95.8       94.2     62.17
> 60.63     41.16    40.24
>
> - So, all the times (including the total execution time that I'm not
> showing here) are the same between PetscLogStage and the std::chrono clock,
> except for the part of MatCreate. Maybe that part is very unbalanced?
>

MatCreate() does nothing at all, but it does have a synchronization (to
check the comm). So you must be very imbalanced
_coming into_ MatCreate. It also appears that 0 is more imbalanced than the
rest, so maybe you are doing serial work on
0 that no one else does before you call MatCreate.

  Matt


> - The time of the MatCreate given by the PetscLogStage is not changing.
>
> Ale
>
> El vie., 8 mar. 2019 a las 17:00, Jed Brown (<jed at jedbrown.org>) escribió:
>
>> This is very unusual.  MatCreate() does no work, merely dup'ing a
>> communicator (or referencing an inner communicator if this is not the
>> first PetscObject on the provided communicator).  What size matrices are
>> you working with?  Can you send some performance data and (if feasible)
>> a reproducer?
>>
>> Ale Foggia via petsc-users <petsc-users at mcs.anl.gov> writes:
>>
>> > Hello all,
>> >
>> > I have a problem with the scaling of the MatCreate() function. I wrote a
>> > code to diagonalize sparse matrices and I'm running it in parallel. I've
>> > observed a very bad speedup of the code and it's given by the MatCreate
>> > part of it: for a fixed matrix size, when I increase the number of
>> > processes the time taken by the function also increases. I wanted to
>> know
>> > if you expect this behavior or if maybe there's something wrong with my
>> > code. When I go to (what I consider) very big matrix sizes, and
>> depending
>> > on the number of mpi processes, in some cases, MatCreate takes more time
>> > than the time the solver takes to solve the system for one eigenvalue or
>> > the time it takes to set up the values.
>> >
>> > Ale
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190311/0a8b5309/attachment.html>