[petsc-users] scaling a parallel matrix

[Barry Smith]

On Sep 23, 2011, at 1:48 PM, Jed Brown wrote:

> On Fri, Sep 23, 2011 at 13:35, khalid ashraf <khalid_eee at yahoo.com> wrote:
> Hi, I have a vector defined on a 3D DA of size ~1000X1000X30. I run the program on ~4000 processors. 
> After doing the calculations, print the vector in vtk format. I need to print about ~10 times at different time steps.
> The printing takes about 3 hours. Is there a way to reduce the time required to print parallel vector printing.
> 
> The VTK ASCII format is a horrible parallel format. One reliable option if you like the VTK tool chain is to write a "parallel" VTK XML format with binary appended data. You can also write an HDF5 file and either provide your own reader for whatever visualization software you are using or write an XDMF file to inform the reader of what to find where.
>  
> 
> I am trying to average over some blocks of grid points and represent the data with a smaller sized vector defined on a smaller 3D DA. One thing I am not clear is that how to get the same layout for the large and smaller DA, so that all the calculations are local when doing the averaging. Could you please provide any guidance ?
> 
> See the lx,ly,lz parameters to DMDACreate3d().
> 
> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/DM/DMDACreate3d.html
 
  Or you can generate the finer DMDA by refining the coarse grid with DMRefine() or generate the coarser mesh by DMCoarsening the finer. They will automatically have the correct layout. Note you can use DMGetInterpolation() to get an operator that can be used to restrict from the fine grid to the coarse then scale by the vector argument returned by DMGetInterpolation(). Using the MatRestrict() with that operator performs a local piecewise linear averaging.

   Barry