Matrix Allocation

[Andrew Colombi]

>  1) Could you be calling MatGetInfo() with the wrong matrix?

Nice try, but I only have one matrix ;-)  I call it A.  We get along.

>  2) Can you use -info to check the numbers?
>  3) Can you try MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, 1)?

Yes and yes.  In fact, I even wrote a small test app just for this
purpose which is revealing some interesting behavior.  Results:
Additional memory is being malloced; however, MatGetInfo does not
appear to be reporting this.  Here I've grepped the output for
"malloc":

$ rjq ./make -info | grep malloc
[1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 1
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 1
[ 1]    mallocs 0 memory 64
[ 0]    mallocs 0 memory 64

The initial messages are from -info, and the last two are from my
MatGetInfo snippet.  Clearly two mallocs are happening that are not
being reported.  Maybe this is reason: those mallocs don't happen in
MatAssembly_MPIAIJ?  The reason for this is that each processor only
computes the rows it's reponsible for.

Thoughts?
-Andrew

On Mon, Jun 30, 2008 at 1:38 PM, Matthew Knepley <knepley at gmail.com> wrote:
>  1) Could you be calling MatGetInfo() with the wrong matrix?
>
>  2) Can you use -info to check the numbers?
>
>  3) Can you try MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, 1)?
>
>     Matt
>
> On Mon, Jun 30, 2008 at 12:25 PM, Andrew Colombi <acolombi at gmail.com> wrote:
>> I am a little confused by some behavior.
>>
>> I've been trying to understand some poor performance numbers and I
>> noticed that if I doubled the preallocation numbers (presently I'm
>> setting d_nz and o_nz to constants, no effort has been made to get the
>> numbers _right_) performance improved dramatically (about 20X).
>>
>> This isn't so surprising except for the report from MatGetInfo.  Even
>> with the lower preallocation numbers I'm getting 0 in info.mallocs for
>>
>> MatInfo info;
>> MatGetInfo(pc_A, MAT_LOCAL, &info); // done and reported on each
>> machine individually
>>
>> So, my question is.... well.. why?  If I'm seeing 0 mallocs before
>> doubling prealloation shouldn't that mean I've preallocated enough?
>> Or are their some switches I need to use to enable malloc counting?
>> Also you can see (according to the same call to MetGetInfo) I'm
>> wasting a lot of memory:
>>
>> // after doubling preallocation
>> nz_alloc 6.2704e+07 nz_used 1.9125e+07 nz_unneed 4.3579e+07
>>
>> (these are print outs of info.nz_allocated, info.nz_used and info.nzunneeded).
>>
>> Any thoughts?  For now memory is not a bottleneck, so I guess I'll be
>> satisfied with guessing big numbers for d_nz and o_nz.  Still, I spent
>> a lot of time scratching my head since guessing higher numbers didn't
>> seem likely to have an effect.
>>
>> Thanks,
>> -Andrew
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
>
>