[petsc-users] help me to solve 6-dim PDE with PETSc with advice

[Barry Smith]

> On Jul 29, 2017, at 7:01 PM, Oleksandr Koshkarov <olk548 at mail.usask.ca> wrote:
> 
> Thank you for the response,
> 
> Now that you said about additional communication for fft, I want to ask another question:
> 
> What if I disregard FFT and will compute convolution directly, I will rewrite it as vector-matrix-vector product
> 
> x^T A x (where T - is transpose, x - is my state vector, and A is a matrix which represents convolution and probably it will be not very sparse).
> 
> Yes - I will do more work (alghorithm will be slower), but will I win if we take the parallelization into account - less communication + probably more scalable algorithm?

   This is completely a guess. No you won't win, the FFT is just so amazing not using it will inflate the work so much there is no recovery.


> 
> Best regards,
> 
> Alex.
> 
> 
> On 07/29/2017 05:23 PM, Barry Smith wrote:
>>   DMDA provides a simple way to do domain decomposition of structured grid meshes in the x, y, and z direction, that is it makes it easy to efficiently chop up vectors in all 3 dimensions, allowing very large problems easily. Unfortunately it only goes up to 3 dimensions and attempts to produce versions for higher dimensions have failed.
>> 
>>   If you would like to do domain decomposition across all six of your dimensions (which I think you really need to do to solve large problems) we don't have a tool that helps with doing the domain decomposition. To make matters more complicated, likely the 3d mpi-fftw that need to be applied require rejiggering the data across the nodes to get it into a form where the ffts can be applied efficiently.
>> 
>>    I don't think there is necessarily anything theoretically difficult about managing the six dimensional domain decomposition I just don't know of any open source software out there that helps to do this. Maybe some PETSc users are aware of such software and can chime in.
>> 
>>    Aside from these two issues :-) PETSc would be useful for you since you could use our time integrators and nonlinear solvers. Pulling out subvectors to process on can be done with with either VecScatter or VecStrideScatter, VecStrideGather etc depending on the layout of the unknowns, i.e. how the domain decomposition is done.
>> 
>>     I wish I had better answers for managing the 6d domain decomposition
>> 
>>    Barry
>> 
>> 
>> 
>> 
>>> On Jul 29, 2017, at 5:06 PM, Oleksandr Koshkarov <olk548 at mail.usask.ca> wrote:
>>> 
>>> Dear All,
>>> 
>>> I am a new PETSc user and I am still in the middle of the manual (I have finally settled to choose PETSc as my main numerical library) so I am sorry beforehand if my questions would be naive.
>>> 
>>> I am trying to solve 6+1 dimensional Vlasov equation with spectral methods. More precisely, I will try to solve half-discretized equations of the form (approximate form) with pseudospectral Fourier method:
>>> 
>>> (Equations are in latex format, the nice website to see them is https://www.codecogs.com/latex/eqneditor.php)
>>> 
>>> \frac{dC_{m,m,p}}{dt} =\\
>>> \partial_x \left ( a_n C_{n+1,m,p} +b_n C_{n,m,p} +c_n C_{n-1,m,p} \right ) \\
>>> + \partial_y \left ( a_m C_{n,m+1,p} +b_m C_{n,m,p} +c_m C_{n,m-1,p}  \right ) \\
>>> + \partial_z \left ( a_p C_{n,m,p+1} +b_p C_{n,m,p} +c_p C_{n,m,p-1}  \right ) \\
>>> + d_n E_x C_{n-1,m,p} + d_m E_x C_{n,m-1,p} + d_p E_x C_{n,m,p-1} \\
>>> + B_x (e_{m,p} C_{n,m-1,p-1} + f_{m,p}C_{n,m-1,p+1} + \dots) + B_y (\dots) + B_z (\dots)
>>> 
>>> 
>>> where a,b,c,d,e,f are some constants which can depend on n,m,p,
>>> 
>>> n,m,p = 0...N,
>>> 
>>> C_{n,m,p} = C_{n,m,p}(x,y,z),
>>> 
>>> E_x = E_x(x,y,z), (same for E_y,B_x,...)
>>> 
>>> and fields are described with equation of the sort (linear pdes with source terms dependent on C):
>>> 
>>> \frac{dE_x}{dt} = \partial_y B_z - \partial_z B_x + (AC_{1,0,0} + BC_{0,0,0}) \\
>>> \frac{dB_x}{dt} = -\partial_y E_z + \partial_z E_x
>>> 
>>> with A,B being some constants.
>>> 
>>> I will need fully implicit Crank–Nicolson method, so my plan is to use SNES or TS where
>>> 
>>> I will use one MPI PETSc vector which describes the state of the system (all, C, E, B),  then I can evolve linear part with just matrix multiplication.
>>> 
>>> The first question is, should I use something like DMDA? if so, it is only 3dimensional but I need 6 dimensional vectots? Will it be faster then matrix multiplication?
>>> 
>>> Then to do nonlinear part I will need 3d mpi-fftw to compute convolutions. The problem is, how do I extract subvectors from full big vector state? (what is the best approach?)
>>> 
>>> If you have some other suggestions, please feel free to share
>>> 
>>> Thanks and best regards,
>>> 
>>> Oleksandr Koshkarov.
>>> 
>