[petsc-users] Parallelizing a matrix-free code

[Matthew Knepley]

On Tue, Oct 17, 2017 at 4:21 AM, Michael Werner <michael.werner at dlr.de>
wrote:

> That's something I'm still struggling with. In the serial case, I can
> simply extract the values from the original grid, and since the ordering of
> the Jacobian is the same there is no problem. In the parallel case this is
> still a more or less open question. That's why I thought about reordering
> the Jacobian. As long as the position of the individual IDs is the same for
> both, I don't have to care about their absolute position.
>
> I also wanted to thank you for your previous answer, it seems that the
> application ordering might be what I'm looking for. However, in the
> meantime I stumbled about another problem, that I have to solve first. My
> new problem is, that I call the external code within the shell matrix'
> multiply call. But in a parallel case, this call obviously gets called once
> per process. So right now I'm trying to circumvent this, so it might take a
> while before I'm able to come back to the original problem...
>

I am not understanding. Is your original code parallel?

  Thanks,

     Matt


> Kind regards,
> Michael
>
> Am 16.10.2017 um 17:25 schrieb Praveen C:
>
> I am interested to learn more about how this works. How are the vectors
> created if the ids are not contiguous in a partition ?
>
> Thanks
> praveen
>
> On Mon, Oct 16, 2017 at 2:02 PM, Stefano Zampini <
> stefano.zampini at gmail.com> wrote:
>
>>
>>
>> 2017-10-16 10:26 GMT+03:00 Michael Werner <michael.werner at dlr.de>:
>>
>>> Hello,
>>>
>>> I'm having trouble with parallelizing a matrix-free code with PETSc. In
>>> this code, I use an external CFD code to provide the matrix-vector product
>>> for an iterative solver in PETSc. To increase convergence rate, I'm using
>>> an explicitly stored Jacobian matrix to precondition the solver. This works
>>> fine for serial runs. However, when I try to use multiple processes, I face
>>> the problem that PETSc decomposes the preconditioner matrix, and probably
>>> also the shell matrix, in a different way than the external CFD code
>>> decomposes the grid.
>>>
>>> The Jacobian matrix is built in a way, that its rows and columns
>>> correspond to the global IDs of the individual points in my CFD mesh
>>>
>>> The CFD code decomposes the domain based on the proximity of points to
>>> each other, so that the resulting subgrids are coherent. However, since its
>>> an unstructured grid, those subgrids are not necessarily made up of points
>>> with successive global IDs. This is a problem, since PETSc seems to
>>> partition the matrix in  coherent slices.
>>>
>>> I'm not sure what the best approach to this problem might be. Is it
>>> maybe possible to exactly tell PETSc, which rows/columns it should assign
>>> to the individual processes?
>>>
>>>
>> If you are explicitly setting the values in your Jacobians via
>> MatSetValues(), you can create a ISLocalToGlobalMapping
>>
>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/
>> IS/ISLocalToGlobalMappingCreate.html
>>
>> that maps the numbering you use for the Jacobians to their counterpart in
>> the CFD ordering, then call MatSetLocalToGlobalMapping and then use
>> MatSetValuesLocal with the same arguments you are calling MatSetValues now.
>>
>> Otherwise, you can play with the application ordering
>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/AO/index.html
>>
>>
>>
>>
>> --
>> Stefano
>>
>
>
> --
>
> ____________________________________________________
>
> Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR)
> Institut für Aerodynamik und Strömungstechnik | Bunsenstr. 10 | 37073 Göttingen <https://maps.google.com/?q=Bunsenstr.+10+%7C+37073+G%C3%B6ttingen&entry=gmail&source=g>
>
> Michael Werner
> Telefon 0551 709-2627 | Telefax 0551 709-2811 | Michael.Werner at dlr.de
> DLR.de
>
>
>
>
>
>
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171017/8e7604e9/attachment.html>