[petsc-users] Parallelizing a matrix-free code

[Michael Werner]

That's something I'm still struggling with. In the serial case, I can 
simply extract the values from the original grid, and since the ordering 
of the Jacobian is the same there is no problem. In the parallel case 
this is still a more or less open question. That's why I thought about 
reordering the Jacobian. As long as the position of the individual IDs 
is the same for both, I don't have to care about their absolute position.

I also wanted to thank you for your previous answer, it seems that the 
application ordering might be what I'm looking for. However, in the 
meantime I stumbled about another problem, that I have to solve first. 
My new problem is, that I call the external code within the shell 
matrix' multiply call. But in a parallel case, this call obviously gets 
called once per process. So right now I'm trying to circumvent this, so 
it might take a while before I'm able to come back to the original 
problem...

Kind regards,
Michael

Am 16.10.2017 um 17:25 schrieb Praveen C:
> I am interested to learn more about how this works. How are the 
> vectors created if the ids are not contiguous in a partition ?
>
> Thanks
> praveen
>
> On Mon, Oct 16, 2017 at 2:02 PM, Stefano Zampini 
> <stefano.zampini at gmail.com <mailto:stefano.zampini at gmail.com>> wrote:
>
>
>
>     2017-10-16 10:26 GMT+03:00 Michael Werner <michael.werner at dlr.de
>     <mailto:michael.werner at dlr.de>>:
>
>         Hello,
>
>         I'm having trouble with parallelizing a matrix-free code with
>         PETSc. In this code, I use an external CFD code to provide the
>         matrix-vector product for an iterative solver in PETSc. To
>         increase convergence rate, I'm using an explicitly stored
>         Jacobian matrix to precondition the solver. This works fine
>         for serial runs. However, when I try to use multiple
>         processes, I face the problem that PETSc decomposes the
>         preconditioner matrix, and probably also the shell matrix, in
>         a different way than the external CFD code decomposes the grid.
>
>         The Jacobian matrix is built in a way, that its rows and
>         columns correspond to the global IDs of the individual points
>         in my CFD mesh
>
>         The CFD code decomposes the domain based on the proximity of
>         points to each other, so that the resulting subgrids are
>         coherent. However, since its an unstructured grid, those
>         subgrids are not necessarily made up of points with successive
>         global IDs. This is a problem, since PETSc seems to partition
>         the matrix in  coherent slices.
>
>         I'm not sure what the best approach to this problem might be.
>         Is it maybe possible to exactly tell PETSc, which rows/columns
>         it should assign to the individual processes?
>
>
>     If you are explicitly setting the values in your Jacobians via
>     MatSetValues(), you can create a ISLocalToGlobalMapping
>
>     http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/IS/ISLocalToGlobalMappingCreate.html
>     <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/IS/ISLocalToGlobalMappingCreate.html>
>
>     that maps the numbering you use for the Jacobians to their
>     counterpart in the CFD ordering, then call
>     MatSetLocalToGlobalMapping and then use MatSetValuesLocal with the
>     same arguments you are calling MatSetValues now.
>
>     Otherwise, you can play with the application ordering
>     http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/AO/index.html
>     <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/AO/index.html>
>
>
>
>     -- 
>     Stefano
>
>

-- 

____________________________________________________

Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR)
Institut für Aerodynamik und Strömungstechnik | Bunsenstr. 10 | 37073 Göttingen

Michael Werner
Telefon 0551 709-2627 | Telefax 0551 709-2811 | Michael.Werner at dlr.de
DLR.de









-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171017/c9437a5b/attachment.html>