<div dir="ltr"><div dir="ltr">On Thu, Dec 17, 2020 at 7:00 AM Roland Richter <<a href="mailto:roland.richter@ntnu.no">roland.richter@ntnu.no</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  

    
  
  <div>
    <p>Dear all,</p>
    <p>I wanted to use PetscArraycpy for copying a part of one complex
      matrix A with a row length of a_len and an offset of a_off into
      another matrix B with a row length of b_len (smaller than a_len -
      a_off), using the following code snippet:</p>
    <p><i>        PetscScalar *A_ptr, *B_ptr;</i><i><br>
      </i><i>        MatDenseGetArray(A, &A_ptr);</i><i><br>
      </i><i>        MatDenseGetArray(B, &B_ptr);</i><i><br>
      </i><i>        MatView(A, PETSC_VIEWER_STDOUT_WORLD);</i><i><br>
      </i><i>        for(size_t i = 0; i < num_local_rows; ++i) {</i><i><br>
      </i><i>            PetscArraycpy(B_ptr + i * b_len, (2 * a_off +
        A_ptr) + i * a_len, b_len);</i><i><br>
      </i><i>        }</i></p>
    <p><i>        MatAssemblyBegin(B, MAT_FINAL_ASSEMBLY);</i><i><br>
      </i><i>        MatAssemblyEnd(B, MAT_FINAL_ASSEMBLY);</i><i><br>
      </i><i>        MatView(B, PETSC_VIEWER_STDOUT_WORLD);</i></p>
    <p>When printing the first row of matrix A (with a_len = 128, a_off
      = 76 and b_len = 26), I get <br>
    </p>
    <p><i>0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        -7.5118186821231378e-02 + -1.2515848507502547e-01i
        5.7593629917958706e+00 + 1.6535197175842331e+00i
        -6.3062119866941906e+01 + 3.2118985283369987e+01i
        1.6228535942636518e+02 + -4.4588492144691378e+02i
        8.7350162264986420e+02 + 2.0568440963147814e+03i
        -7.4258479521622921e+03 + -3.3031631388498363e+03i
        2.2699374989663269e+04 + -7.8289291098031481e+03i
        -2.7846379282467926e+04 + 5.2456793075148809e+04i
        -3.2554674832896777e+04 + -1.2108819252524960e+05i
        1.9430868047197413e+05 + 1.2114559011378702e+05i
        -3.5831799834334152e+05 + 7.0086227392363056e+04i
        3.0028983479603863e+05 + -4.1447894788669585e+05i
        7.8224949502036819e+04 + 6.2926756374162808e+05i
        -5.3474873053744854e+05 + -4.4718914789259754e+05i
        6.8111038372267899e+05 + -3.6947593166740131e+04i
        -4.1287212326113920e+05 + 4.2925417635846150e+05i
        7.1098224367113344e+03 + -4.6490743916366581e+05i
        2.1807010096419850e+05 + 2.4106178223450572e+05i
        -2.0304162108015743e+05 + -1.7254769976182859e+04i
        9.0164628688356752e+04 + -7.0830186001321214e+04i
        -8.9050769071193699e+03 + 5.7267241933255813e+04i
        -1.4789632694550470e+04 + -2.1786332309775924e+04i
        1.0491004489879153e+04 + 2.3873712516742830e+03i
        -3.4183915782335853e+03 + 1.9886861075931499e+03i
        3.7807432692260045e+02 + -1.2521184263406540e+03i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        -0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i</i></p>
    <p>and for the first row of B I get <br>
    </p>
    <p><i>-0.0000000000000000e+00 + 0.0000000000000000e+00i
        -7.5118186821231378e-02 + -1.2515848507502547e-01i
        5.7593629917958706e+00 + 1.6535197175842331e+00i
        -6.3062119866941906e+01 + 3.2118985283369987e+01i
        1.6228535942636518e+02 + -4.4588492144691378e+02i
        8.7350162264986420e+02 + 2.0568440963147814e+03i
        -7.4258479521622921e+03 + -3.3031631388498363e+03i
        2.2699374989663269e+04 + -7.8289291098031481e+03i
        -2.7846379282467926e+04 + 5.2456793075148809e+04i
        -3.2554674832896777e+04 + -1.2108819252524960e+05i
        1.9430868047197413e+05 + 1.2114559011378702e+05i
        -3.5831799834334152e+05 + 7.0086227392363056e+04i
        3.0028983479603863e+05 + -4.1447894788669585e+05i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        6.9004764357806349e-310 + 6.9004764357806349e-310i
        6.9004764365446580e-310 + 6.9004764300000669e-310i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i
        0.0000000000000000e+00 + 0.0000000000000000e+00i</i></p>
    <p>Apparently, only 13 complex values have been copied, and not 26.
      Moreover, if my source destination is chosen to be (a_off + A_ptr)
      instead, I will just copy 0-values. When increasing the number of
      values I would like to copy, nothing changes (except getting a
      segfault for sufficient large values).</p>
    <p>Why does that happen? And how can I copy all values into the
      second matrix, and not only half of them?</p></div></blockquote><div>Can you make a minimal example? It looks like it should work. If I can run the code, I can make it work. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
    <p>Another question: Is there a parallel version of that function,
      to copy all local rows in parallel, or do I have to write it
      myself, for example by using OpenMP?</p></div></blockquote><div>The copy should be vectorized by the compiler. If you have idle cores waiting for something, you could possibly use OpenMP. However,  as Jed points out,<br></div><div>the time to fork and join is likely to exceed your speedup.</div><div><br></div><div>  Thanks,</div><div><br></div><div>     Matt</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
    <p>Thanks!</p>
    <p>Roland<br>
    </p>
    <p><br>
    </p>
    <p><br>
    </p>
  </div>

</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>