[petsc-users] MatCopy and MatSetValue consume most 99 percentage of runtime

[Barry Smith]

On Apr 19, 2011, at 10:08 PM, Vijay S. Mahadevan wrote:

>>   Don't call MatCopy() on anything that you have NOT fully filled up, same with MatDiagonalSet() and MatZeroEntries(). You should only do those operations on matrices that you have filled up and called MatAssembly on already. You can use MatDiagonalSet() directly on a naked matrix but only if you are not putting other values in.
> 
> Barry, just to confirm, are you saying that MatZeroEntries would
> nullify the preallocation completely if called before
> AssmeblyBegin/End ? I have been doing this quite often before a linear
> system assembly and have not noticed extra mallocs during the process.
> Is there something that I am misunderstanding in the above statement ?

   My mistake. Yes if you call MatZeroEntries() on a matrix you have not started putting values in it will not destroy the preallocation information. 

   Barry

> I would much appreciate if you can clarify.
> 
> Thanks,
> Vijay
> 
> On Tue, Apr 19, 2011 at 9:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> On Apr 19, 2011, at 8:50 PM, Debao Shao wrote:
>> 
>>> Here is my sample code:
>>>  ierr = MatZeroEntries( M ); assert( ierr  == 0);
>>>  ierr = MatDiagonalSet( M, vec.pv, INSERT_VALUES );
>>>  ierr = MatCopy( ms->M, mStorage->M,  DIFFERENT_NONZERO_PATTERN );
>>> 
>>> I checked PETSC api, both "MatDiagonalSet" and "MatCopy" called MatAssembly***, Is the usage wrong, or, how to deal with the problem?
>> 
>>   Don't call MatCopy() on anything that you have NOT fully filled up, same with MatDiagonalSet() and MatZeroEntries(). You should only do those operations on matrices that you have filled up and called MatAssembly on already. You can use MatDiagonalSet() directly on a naked matrix but only if you are not putting other values in.
>> 
>>    Barry
>> 
>> I know these "rules" may seem strange but the problem is that these operations need to know the nonzero pattern of the sparse matrix and since you haven't set anything much in them yet they have to assume it is empty and blast away the preallocation information.  Basically you shouldn't do much in the way of operations on matrices until you've fully assembled them.
>> 
>>> 
>>> Thanks,
>>> Debao
>>> -----Original Message-----
>>> From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith
>>> Sent: Wednesday, April 20, 2011 9:40 AM
>>> To: PETSc users list
>>> Subject: Re: [petsc-users] MatCopy and MatSetValue consume most 99 percentage of runtime
>>> 
>>> 
>>> On Apr 19, 2011, at 8:31 PM, Debao Shao wrote:
>>> 
>>>> Hi, Barry:
>>>> 
>>>> Thanks for the reply.
>>>> 
>>>> I preallocated enough space for the sparse matrix, but I found mat->data->imax changed after "MatAssemblyEnd", and it caused many rowmax less than the number of nonzeros per row, then "MatSeqXAIJReallocateAIJ" is called frequently when doing MatSetValues again to the matrix.
>>> 
>>>   Are you using MatZeroRows()? If so call MatSetOption(mat, MAT_KEEP_NONZERO_PATTERN) before calling the zero rows to retain that structure.
>>> 
>>>    If you are not using MatZeroRows() then apparently the first time you set values in there and call MatAssemblyEnd() you have left many locations that later will be filled unfilled and so they are eliminated at MatAssembly time.  You must make sure that all potentially nonzero locations get a value put in initially (put zero for the locations that you don't yet have  a value for) before you first call MatAssemblyEnd().
>>> 
>>>   Barry
>>> 
>>> PETSc matrices have no way of retaining extra locations you preallocated for unless you put something (like 0) in there.
>>> 
>>>> 
>>>> Any suggestions?
>>>> 
>>>> Thanks,
>>>> Debao
>>>> -----Original Message-----
>>>> From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith
>>>> Sent: Friday, April 15, 2011 9:25 PM
>>>> To: PETSc users list
>>>> Subject: Re: [petsc-users] MatCopy and MatSetValue consume most 99 percentage of runtime
>>>> 
>>>> 
>>>>  Debao,
>>>> 
>>>>     Please see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly It should resolve the difficulties.
>>>> 
>>>>   Barry
>>>> 
>>>> On Apr 15, 2011, at 2:16 AM, Debao Shao wrote:
>>>> 
>>>>> Dear Petsc:
>>>>> 
>>>>> I'm trying on Petsc iterative solver(KSPCG & PCJACOBI), but it's strange that the two functions "MatCopy" and "MatSetValue" consume most of runtime, and the functions were not called frequently, just several times.
>>>>> 
>>>>> My libpetsc.a is built as follows:
>>>>> 1, /config/configure.py --with-mpi=0 --with-debugging=0 -with-log=0 -with-info=0
>>>>> 2, make all;
>>>>> 
>>>>> It's very appreciated to get your reply.
>>>>> 
>>>>> Thanks a lot,
>>>>> Debao
>>>>> 
>>>>> -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt.
>>>> 
>>>> 
>>>> -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt.
>>> 
>>> 
>>> -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt.
>> 
>>