[petsc-users] Preallocation woes again

[Matthew Knepley]

On Fri, Jan 27, 2012 at 12:04 PM, Tabrez Ali <stali at geology.wisc.edu> wrote:

> Matt
>
> Thanks. Yes I can wait for a few weeks/months (its only a local memory
> footprint issue and in no way hampers my ability to solve problems).
>
> Btw just for curiosity are you using the same technique discussed by Barry?
>

Basically. You have to store indices while counting if you don't want to
overcount. This is the part he glosses over.

   Matt


> Maybe I am overlooking something trivial in his post.
>
> Tabrez
>
>
> On Jan 27, 2012, at 10:26 AM, Matthew Knepley wrote:
>
> On Fri, Jan 27, 2012 at 10:32 AM, Tabrez Ali <stali at geology.wisc.edu>wrote:
>
>> PETSc Gurus
>>
>> First I want to thank you for patiently answering questions that I have
>> asked in past regarding preallocation. Unfortunately I am still having
>> problems.
>>
>> I have a small unstructured FE (elasticity) code that uses PETSc.
>>
>> Unfortunately I am not yet able to find an _efficient_ way of calculating
>> the non-zero structure so I simply overestimate the o_nz and d_nz values (=
>> dimension x max_number_of nodes _a_node_contributes_to) which means that my
>> stiffness matrix consumes at least 2X more memory. For example, for a
>> perfectly structured mesh of linear hexes this number is 3*27=81=o_nz=d_nz.
>> In complicated 3D linear tet/hex meshes that I have generated I rarely need
>> to set a value greater than 150. In general I have found that as long as I
>> keep DOF/core between 100-200K (assuming 1GB/per core) there is enough
>> local RAM left even when memory for stiffness matrix is overestimated by
>> 2X-6X.
>>
>> In any case now I do want to preallocate exactly for better memory
>> performance and the ambiguity involved in choosing a reasonable o_nz/d_nz.
>> The way I am trying to do it involves loops like ...
>>
>> do i=1, num_local_elements
>>   do j=1, num_total_nodes
>>      ...
>>   end do
>> end do
>>
>> or using lists that involve searches.
>>
>> This unfortunately takes much time (much more than assembly and solve)
>> due to the second loop. I am aware of many posts/slides by PETSc authors
>> that mention that the non-zero structure can be found simply by looping
>> once through elements but what to do next is not quite clear.
>>
>> I am also aware of this post by Barry [ http://lists.mcs.anl.gov/**
>> pipermail/petsc-users/2008-**May/003020.html<http://lists.mcs.anl.gov/pipermail/petsc-users/2008-May/003020.html>] but I cannot get it to work for a simple 4 element problem. Can some one
>> please expand on the strategy being suggested by Barry specially where the
>> loops are discussed (I understand the arrays and the vecscatter part). I am
>> also not sure how does using a t of 0.5 and 1 prevents double counting.
>> Shouldn't it be 0 and 1?
>>
>> I am also aware of some capabilities in DMMESH but right now I dont
>> understand it well enough to utilize it.
>>
>
> I agree with you here. Its not easy enough to use. However, I have
> rewritten the basics of it completely in C, which
> will all have easy Fortran bindings. You can specify your mesh with
> adjacency lists using Fortran arrays. The last part
> I am working on is matrix preallocation. I would not normally ask you to
> wait, but I think I am pretty close. Mail me
> directly if you want more info. I think it should only take me a few weeks
> to finish.
>
>   Thanks,
>
>      Matt
>
> Alternatively does PETSc support MATMPIAIJ of size=integer(1)/logical
>> instead of real(8) which I can use to put zeros/ones (when I loop over
>> elements the first time) to get the non-zero structure?
>>
>> Thanks in advance.
>>
>> Tabrez
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120127/ce1b5ba6/attachment.htm>