<div dir="ltr"><div dir="ltr">On Thu, Dec 17, 2020 at 7:00 AM Roland Richter <<a href="mailto:roland.richter@ntnu.no">roland.richter@ntnu.no</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Dear all,</p>
<p>I wanted to use PetscArraycpy for copying a part of one complex
matrix A with a row length of a_len and an offset of a_off into
another matrix B with a row length of b_len (smaller than a_len -
a_off), using the following code snippet:</p>
<p><i> PetscScalar *A_ptr, *B_ptr;</i><i><br>
</i><i> MatDenseGetArray(A, &A_ptr);</i><i><br>
</i><i> MatDenseGetArray(B, &B_ptr);</i><i><br>
</i><i> MatView(A, PETSC_VIEWER_STDOUT_WORLD);</i><i><br>
</i><i> for(size_t i = 0; i < num_local_rows; ++i) {</i><i><br>
</i><i> PetscArraycpy(B_ptr + i * b_len, (2 * a_off +
A_ptr) + i * a_len, b_len);</i><i><br>
</i><i> }</i></p>
<p><i> MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);</i><i><br>
</i><i> MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);</i><i><br>
</i><i> MatView(B, PETSC_VIEWER_STDOUT_WORLD);</i></p>
<p>When printing the first row of matrix A (with a_len = 128, a_off
= 76 and b_len = 26), I get <br>
</p>
<p><i>0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
-7.5118186821231378e-02 + -1.2515848507502547e-01i
5.7593629917958706e+00 + 1.6535197175842331e+00i
-6.3062119866941906e+01 + 3.2118985283369987e+01i
1.6228535942636518e+02 + -4.4588492144691378e+02i
8.7350162264986420e+02 + 2.0568440963147814e+03i
-7.4258479521622921e+03 + -3.3031631388498363e+03i
2.2699374989663269e+04 + -7.8289291098031481e+03i
-2.7846379282467926e+04 + 5.2456793075148809e+04i
-3.2554674832896777e+04 + -1.2108819252524960e+05i
1.9430868047197413e+05 + 1.2114559011378702e+05i
-3.5831799834334152e+05 + 7.0086227392363056e+04i
3.0028983479603863e+05 + -4.1447894788669585e+05i
7.8224949502036819e+04 + 6.2926756374162808e+05i
-5.3474873053744854e+05 + -4.4718914789259754e+05i
6.8111038372267899e+05 + -3.6947593166740131e+04i
-4.1287212326113920e+05 + 4.2925417635846150e+05i
7.1098224367113344e+03 + -4.6490743916366581e+05i
2.1807010096419850e+05 + 2.4106178223450572e+05i
-2.0304162108015743e+05 + -1.7254769976182859e+04i
9.0164628688356752e+04 + -7.0830186001321214e+04i
-8.9050769071193699e+03 + 5.7267241933255813e+04i
-1.4789632694550470e+04 + -2.1786332309775924e+04i
1.0491004489879153e+04 + 2.3873712516742830e+03i
-3.4183915782335853e+03 + 1.9886861075931499e+03i
3.7807432692260045e+02 + -1.2521184263406540e+03i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
-0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i</i></p>
<p>and for the first row of B I get <br>
</p>
<p><i>-0.0000000000000000e+00 + 0.0000000000000000e+00i
-7.5118186821231378e-02 + -1.2515848507502547e-01i
5.7593629917958706e+00 + 1.6535197175842331e+00i
-6.3062119866941906e+01 + 3.2118985283369987e+01i
1.6228535942636518e+02 + -4.4588492144691378e+02i
8.7350162264986420e+02 + 2.0568440963147814e+03i
-7.4258479521622921e+03 + -3.3031631388498363e+03i
2.2699374989663269e+04 + -7.8289291098031481e+03i
-2.7846379282467926e+04 + 5.2456793075148809e+04i
-3.2554674832896777e+04 + -1.2108819252524960e+05i
1.9430868047197413e+05 + 1.2114559011378702e+05i
-3.5831799834334152e+05 + 7.0086227392363056e+04i
3.0028983479603863e+05 + -4.1447894788669585e+05i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
6.9004764357806349e-310 + 6.9004764357806349e-310i
6.9004764365446580e-310 + 6.9004764300000669e-310i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i
0.0000000000000000e+00 + 0.0000000000000000e+00i</i></p>
<p>Apparently, only 13 complex values have been copied, and not 26.
Moreover, if my source destination is chosen to be (a_off + A_ptr)
instead, I will just copy 0-values. When increasing the number of
values I would like to copy, nothing changes (except getting a
segfault for sufficient large values).</p>
<p>Why does that happen? And how can I copy all values into the
second matrix, and not only half of them?</p></div></blockquote><div>Can you make a minimal example? It looks like it should work. If I can run the code, I can make it work. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>Another question: Is there a parallel version of that function,
to copy all local rows in parallel, or do I have to write it
myself, for example by using OpenMP?</p></div></blockquote><div>The copy should be vectorized by the compiler. If you have idle cores waiting for something, you could possibly use OpenMP. However, as Jed points out,<br></div><div>the time to fork and join is likely to exceed your speedup.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>Thanks!</p>
<p>Roland<br>
</p>
<p><br>
</p>
<p><br>
</p>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>