[petsc-dev] KSP automatic reordering

[Barry Smith]

On Sep 18, 2012, at 8:54 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Tue, Sep 18, 2012 at 8:45 PM, Matthew Knepley <knepley at gmail.com> wrote:
> But it is nasty because it changes the convergence tests, the monitor routines (they report the residual norms in the scale system not the original).  Also, does it unscale the matrix after the solve (or leave it scaled for when it is never going to be used again?). The scaling can screw up algebraic multigrid methods. Does the scaling affect Eisenstat-Walker type convergence tests for Newton's method….  It is nasty code, hard to follow and hard for users to fully appreciate.
> 
> We could do the same ugly hacked up thing for reordering; using a partitioner for between processes and a low-bandwidth ordering within inside the KSPSolve/SetUp().
> 
> I think that becomes too confusing and too intrusive to our code.
>  
> It would be nice to have a cleaner/clearer abstract model in terms of the software layering to handle this.  For example I played with the idea of a "diagonal scaling" PC and "reordering" PC that
> does the change and then has its own KSP inside for the solve. Thus you'd run with -ksp_type preonly -pc_type reorder -reorder_ksp_type gmres -reorder_pc_type ilu etc. But that seems a bit pedantic
> and annoying that you have to put all your "true" solver options with a prefix.
> 
> This is what I had in mind. The thing is, we don't actually want people to use this. Redistributing the matrix is fairly expensive. We want them to change their code to produce a decent ordering. A good way to convince people that it's worthwhile is to put the performance numbers in front of their face. Matrix Market will always do its silly orderings and is stored on disk in a stupid format so that part will always be slow (and generally isn't part of the benchmark result).

   Jed,

      The problem is that users won't fix their code to use the correct ordering; they'll just have it reorder after it is made (since people are lazy), but of course this is true regardless of our interface.

      So should we strip out the diagonal scaling stuff and make it a PC?  

   Barry

>  
> Why not make it part of the matrix? For the minute, assume we are using a DM. Then the
> matrix has the nonzero pattern already. We can use an option to compute a fill-reducing ordering
> and either permute it directly, or just use the permutations on in and out. This insulates it from
> the solver completely.
> 
> At what point is the vector reordered?