<div dir="ltr"><div dir="ltr">On Fri, Dec 4, 2020 at 7:47 AM Roland Richter <<a href="mailto:roland.richter@ntnu.no">roland.richter@ntnu.no</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Ideally those FFTs could be handled in parallel, after they are
not depending on each other. Is that possible with MatFFT, or
should I rather use FFTW for that?</p>
<p></p></div></blockquote><div>Which FFTs? The rows on each process are handled in parallel. For the rows on a process, you could call FFTW yourself to try and get</div><div>instruction level parallelism over rows, but if the rows are long I don't think this would matter much. I guess you could measure. If you</div><div>wanted parallelism among rows, you could divide the matrix more finely. If you wanted parallelism within a row, you would need to change</div><div>the matrix storage format. I am not sure if any packages do this, certainly not any I know of. You could treat the rows as PETSc vectors and</div><div>get parallelism this way I guess.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p>Thanks,</p>
<p>Roland<br>
</p>
<div>Am 04.12.20 um 13:19 schrieb Matthew
Knepley:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Fri, Dec 4, 2020 at 5:32 AM Roland Richter
<<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hei,<br>
<br>
I am currently working on a problem which requires a large
amount of<br>
transformations of a field E(r, t) from time space to
Fourier space E(r,<br>
w) and back. The field is described in a 2d-matrix, with the
r-dimension<br>
along the columns and the t-dimension along the rows.<br>
<br>
For the transformation from time to frequency space and back
I therefore<br>
have to apply a 1d-FFT operation over each row of my matrix.
For my<br>
earlier attempts I used armadillo as matrix library and FFTW
for doing<br>
the transformations. Here I could use fftw_plan_many_dft to
do all FFTs<br>
at the same time. Unfortunately, armadillo does not support
MPI, and<br>
therefore I had to switch to PETSc for larger matrices.<br>
<br>
Based on the examples (such as example 143) PETSc has a way
of doing<br>
FFTs internally by creating an FFT object (using
MatCreateFFT).<br>
Unfortunately, I can not see how I could use that object to
conduct the<br>
operation described above without having to iterate over
each row in my<br>
original matrix (i.e. doing it sequential, not in parallel).<br>
<br>
Ideally I could distribute the FFTs such over my nodes that
each node<br>
takes several rows of the original matrix and applies the
FFT to each of<br>
them. As example, for a matrix with a size of 4x4 and two
nodes node 0<br>
would take row 0 and 1, while node 1 takes row 2 and 3, to
avoid<br>
unnecessary memory transfer between the nodes while
conducting the FFTs.<br>
Is that something PETSc can do, too?<br>
</blockquote>
<div><br>
</div>
<div>The way I understand our setup (I did not write it), we
use plan_many_dft to handle</div>
<div>multiple dof FFTs, but these would be interlaced. You
want many FFTs for non-interlaced</div>
<div>storage, which is not something we do right now. You
could definitely call FFTW directly</div>
<div>if you want.</div>
<div><br>
</div>
<div>Second, above it seems like you just want serial FFTs.
You can definitely create a MatFFT</div>
<div>with PETSC_COMM_SELF, and apply it to each row in the
local rows, or create the plan</div>
<div>yourself for the stack of rows.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Thanks!<br>
<br>
Regards,<br>
<br>
Roland<br>
<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>