[petsc-dev] Fwd: [petsc-maint #57017] Hypre
Barry Smith
bsmith at mcs.anl.gov
Thu Dec 2 08:00:02 CST 2010
Begin forwarded message:
> From: Laurent Michel <laurent.michel at epfl.ch>
> Date: December 2, 2010 5:25:38 AM CST
> To: petsc-maint at mcs.anl.gov
> Subject: Re: [petsc-maint #57017] Hypre
> Reply-To: petsc-maint at mcs.anl.gov, Laurent Michel <laurent.michel at epfl.ch>
>
> Ok. That sounds much easier. So, during the building of the linear
> system, I do something like (when computing the CSR format)
>
> PetscInt start, end; MatGetOwnershipRange(A, &start, &end);
> vector<int> idv, idp; // velocity and pressure indices
> int index = 0, row = 0;
> for (int i = 0; i < nbNodeP1; i++) { // Loop over the nodes of the
> mesh (P1 nodes)
> for (int ndof = 0; ndof < 4; ndof++) { // Loop over the degrees
> of freedom of each node; 3 for v, 1 for p
> if(!nodes[i].is_fixed(ndof)){ // if the node is free, then
> put it in the linear system
> if(start <= row && row < end){ // each process does its
> own part of the job
>
> [stuff for CSR format]
>
> if(0 <= ndof && ndof < 3) // velocity
> idv.push_back(row);
> else // pressure
> idp.push_back(row);
> }
>
> row++;
> }
> }
> }
>
> // conversion to arrays
> int *idp_ptr = new int [idp.size()], *idv_ptr = new int
> [idv.size()]; // fieldsplit indices
> memcpy(idp_ptr, &idp[0], sizeof(int)*idp.size());
> memcpy(idv_ptr, &idv[0], sizeof(int)*idv.size());
>
> // conversion to IS
> IS isv, isp;
> ISCreateGeneral(PETSC_COMM_WORLD, idp.size(), idp_ptr, &isp);
> ISCreateGeneral(PETSC_COMM_WORLD, idv.size(), idv_ptr, &isv);
>
> Then, when I create the preconditioner, I say
>
> PCSetType(pc, PCFIELDSPLIT);
> PCFieldSplitSetIS(pc, isv);
> PCFieldSplitSetIS(pc, isp);
>
> I guess the order matters here. Then, when I call the code, I call it
> among other things with
>
> -pc_fieldsplit_type multiplicative -fieldsplit_0_pc_type ml
> -fieldsplit_1_pc_type jacobi
>
> Ok. Now the thing is, I haven't compiled the ml library, and it seems
> like the petsc servers are down (I can't download anything). Anyway, I
> wanted to try this with something else. Hence, I tried in SEQUENTIAL
>
> -pc_fieldsplit_type multiplicative -fieldsplit_0_pc_type ilu
> -fieldsplit_0_pc_factor_levels 2 -fieldsplit_1_pc_type jacobi
>
> and this works fine; in parallel, with
>
> -pc_fieldsplit_type multiplicative -fieldsplit_0_pc_type asm
> -fieldsplit_1_pc_type jacobi
>
> I get something working. I think I've been able to code this. I'll let
> you know if I encounter some problems with the ml preconditioner (I have
> no idea what it is, so I may have problems).
>
> Regarding all this, I have to mention that until now I've had to reorder
> my matrix elements with
>
> -pc_factor_mat_ordering_type rcm
>
> because it seems like the nodes of my mesh are not numbered
> appropriately. How should I proceed for parallel computations? In
> sequential, I gain about a factor 8 in CPU time doing this.
>
> Thanks,
>
> L.
>
>
>
> Matthew Knepley wrote:
>> On Wed, Dec 1, 2010 at 8:38 AM, Laurent Michel <laurent.michel at epfl.ch
>> <mailto:laurent.michel at epfl.ch>> wrote:
>>
>> Well I'm actually struggling finding a good, scalable
>> preconditioner for
>> my problem. This is one of the problems I have to solve these days.
>>
>> I have spoken to a few people here who are kind of experts in the
>> field
>> of hpc, and I have always got the same answer: the Stokes problem will
>> not scale well. I found some people working on more or less the same
>> problem as I do who actually have a scalable code (sisiphus for
>> instance; I am solving a glacier problem).
>>
>> I wanted to use the fieldsplit preconditioner, I already roughly
>> discussed this with Mr. Knepley, but I'm not quite sure yet how to do
>> this; I mean, I have an unstructured mesh, and I need to define
>> some DAs
>> to make the fieldsplit preconditioner work. It seems complicated.
>>
>>
>> I did not do a good enough job explaining. FieldSplit should not be
>> hard for your
>> problem. You just need to create lists of the unknowns in each field.
>> Each list
>> will be an IS object. Then you call
>>
>> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html
>>
>> once for each field. Then you can independently handle the velocity and
>> pressure preconditioners, and it should produce a scalable solver. I would
>> try -pc_fieldsplit_type multiplicative -fieldsplit_0_pc_type ml
>> -fieldsplit_1_pc_type jacobi
>>
>> Matt
>>
>>
>> In parallel to this, I have seen that decoupling the velocity from the
>> pressure field (with like an Uzawa method or something like that; I'm
>> not an expert here) may be more scalable. I gather this is the same as
>> your fieldsplit preconditioner. I'll have a more thorough look
>> into this
>> if so. There is to my knowledge (which is very small) no documentation
>> related to the handling of unstructured problems with your fieldsplit
>> preconditioner, which I think might be the best option for me. ASM
>> doesn't work well (I have awful results on my cluster; see
>> attachment).
>> I'm solving Ax = b here, with A being the whole finite element matrix,
>> containing the data related to all the fields (velocity and pressure).
>>
>> Any hint on what I could try? It's not central for my PhD thesis, but
>> I'd enjoy some faster code!
>>
>> Regards,
>>
>> L.
>>
>> Barry Smith wrote:
>>> On Dec 1, 2010, at 2:44 AM, Laurent Michel wrote:
>>>
>>>
>>>> Hi,
>>>>
>>>> * OS Version: Linux iacspc3 2.6.31-22-generic #68-Ubuntu SMP
>> Tue Oct
>>>> 26 16:37:17 UTC 2010 x86_64 GNU/Linux
>>>> * PETSc Version: Fri Apr 30 20:23:44 CDT 2010
>>>> * MPI implementation: openmpi
>>>> * Compiler: g++
>>>> * Probable PETSc component: PC
>>>>
>>>> My question is about the use of preconditioner Hypre. I have a
>> colleague
>>>> who uses hypre directly and solves more or less the same kind
>> of problem
>>>> as I do. His code scales pretty well (3D Navier-Stokes) and I
>> wanted to
>>>> try hypre in mine, in order to see what happens. Indeed, for my
>> problem,
>>>> it's known that the algebraic Schwarz method will not scale
>> that good
>>>> (this also what I have noticed during my numerical
>> experiments). Anyway,
>>>> I've tried to run my code with the following additional options:
>>>>
>>>> -pc_type hypre -pc_hypre_type euclid
>>>>
>>>> and it takes forever to solve the linear system (I mean, to
>> apply the
>>>> preconditioner and then actually solve the system).
>>>>
>>>
>>> Likely Euclid is a bad preconditioner for your problem. Run
>> with -log_summary -ksp_converged_reason to completion and then
>> look at the output to see where it is spending all the time. If
>> the time is truely dominated by PCSetUp and or PCApply or requires
>> many many iterations then Euclid is the problem. We've found the
>> Euclid doesn't scale or perform particularlly well.
>>>
>>> If your problem is a "Stokes" like problem that just throwing
>> a generic preconditioner like Euclid at it without doing something
>> special for the fact that it is "Stokes like" problem won't work
>> well. Perhaps Jed has some suggestions.
>>>
>>> Barry
>>>
>>>
>>>> I'm using the MPIAIJ matrix format, with
>> MatMPIAIJSetPreallocationCSR.
>>>> Should I use another format? I've been struggling looking for
>>>> documentation about this. Where can I find any further information?
>>>>
>>>> Regards,
>>>>
>>>> L.
>>>>
>>>> --
>>>> Laurent Michel
>>>> PhD candidate
>>>> EPFL SB IACS ASN
>>>> MA C2 644
>>>> +41 21 693 42 46
>>>> +41 77 433 38 94
>>>>
>>>>
>>>>
>>>
>>> .
>>>
>>>
>>
>>
>> --
>> Laurent Michel
>> PhD candidate
>> EPFL SB IACS ASN
>> MA C2 644
>> +41 21 693 42 46
>> +41 77 433 38 94
>>
>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which
>> their experiments lead.
>> -- Norbert Wiener
>
>
> --
> Laurent Michel
> PhD candidate
> EPFL SB IACS ASN
> MA C2 644
> +41 21 693 42 46
> +41 77 433 38 94
>
>
More information about the petsc-dev
mailing list