[petsc-users] MatCreate performance

[Ale Foggia]

Hello,

I've checked very thoroughly the previous part of the code and I found
where the problem is. As you said, it's due to an imbalance in the previous
part, before MatCreate. Thank you so much for your answers, they helped me
understand.

El lun., 11 mar. 2019 a las 14:37, Mark Adams (<mfadams at lbl.gov>) escribió:

> The PETSc logs print the max time and the ratio max/min.
>
> On Mon, Mar 11, 2019 at 8:24 AM Ale Foggia via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hello all,
>>
>> Thanks for your answers.
>>
>> 1) I'm working with a matrix with a linear size of 2**34, but it's a
>> sparse matrix, and the number of elements different from zero is
>> 43,207,072,74. I know that the distribution of these elements is not
>> balanced between the processes, the matrix is more populated in the middle
>> part.
>>
>> 2) I initialize Slepc. Then I create the basis elements of the system
>> (this part does not involve Petsc/Slepc, and every process is just
>> computing -and owns- an equal amount of basis elements). Then I call:
>> ierr = MatCreate(PETSC_COMM_WORLD, &A); CHKERRQ(ierr);
>> ierr = MatSetType(A, MATMPIAIJ); CHKERRQ(ierr);
>> ierr = MatSetSizes(A, PETSC_DECIDE, PETSC_DECIDE, size, size);
>> CHKERRQ(ierr);
>> ierr = MatMPIAIJSetPreallocation(A, 0, d_nnz, 0, o_nnz); CHKERRQ(ierr);
>> ierr = MatZeroEntries(A); CHKERRQ(ierr);
>> After this, I compute the elements of the matrix and set the values with
>> MatSetValues. The I call EPSSolve (with KrylovSchur and setting the type as
>> EPS_HEP).
>>
>> 3) There are a few more things that are strange to me. I measure the
>> execution time of these parts both with a PetscLogStage and with a
>> std::chrono (in nanoseconds) clock. I understand that the time given by the
>> Log is an average over the processes right? In the case of the std::chrono,
>> I'm only printing the times from process 0 (no average over processes).
>> What I see is the following:
>>                              1024 procs          2048 procs       4096
>> procs        8192 procs
>>                              Log       std          Log       std
>> Log       std        Log       std
>> MatCreate            68.42   122.7      67.08    121.2   62.29    116
>>    73.36    127.4
>> preallocation        140.36  140.3     76.45     76.45   40.31
>> 40.3      21.13     21.12
>> MatSetValues       237.79  237.7     116.6    116.6    60.59    60.59
>> 35.32    35.32
>> ESPSolve             162.8    160        95.8       94.2     62.17
>> 60.63     41.16    40.24
>>
>> - So, all the times (including the total execution time that I'm not
>> showing here) are the same between PetscLogStage and the std::chrono clock,
>> except for the part of MatCreate. Maybe that part is very unbalanced?
>> - The time of the MatCreate given by the PetscLogStage is not changing.
>>
>> Ale
>>
>> El vie., 8 mar. 2019 a las 17:00, Jed Brown (<jed at jedbrown.org>)
>> escribió:
>>
>>> This is very unusual.  MatCreate() does no work, merely dup'ing a
>>> communicator (or referencing an inner communicator if this is not the
>>> first PetscObject on the provided communicator).  What size matrices are
>>> you working with?  Can you send some performance data and (if feasible)
>>> a reproducer?
>>>
>>> Ale Foggia via petsc-users <petsc-users at mcs.anl.gov> writes:
>>>
>>> > Hello all,
>>> >
>>> > I have a problem with the scaling of the MatCreate() function. I wrote
>>> a
>>> > code to diagonalize sparse matrices and I'm running it in parallel.
>>> I've
>>> > observed a very bad speedup of the code and it's given by the MatCreate
>>> > part of it: for a fixed matrix size, when I increase the number of
>>> > processes the time taken by the function also increases. I wanted to
>>> know
>>> > if you expect this behavior or if maybe there's something wrong with my
>>> > code. When I go to (what I consider) very big matrix sizes, and
>>> depending
>>> > on the number of mpi processes, in some cases, MatCreate takes more
>>> time
>>> > than the time the solver takes to solve the system for one eigenvalue
>>> or
>>> > the time it takes to set up the values.
>>> >
>>> > Ale
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190312/cf132cb9/attachment-0001.html>