<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Mar 27, 2015 at 11:07 AM, Klaus Kaiser <span dir="ltr"><<a href="mailto:kaiser@igpm.rwth-aachen.de" target="_blank">kaiser@igpm.rwth-aachen.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    <div>Hallo Matthew,<br>
      <br>
      here is a short example I used for 8 cores:<br>
      <br>
          int d_nnz[3];<br>
          int o_nnz[3];<br>
          d_nnz[0] = 3; d_nnz[1] = 3; d_nnz[2] = 3;<br>
          o_nnz[0] = 6; o_nnz[1] = 6; o_nnz[2] = 6;<br>
          ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0,
      d_nnz, 0, o_nnz,&A);<span class=""><br>
          ierr =
      MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);<br>
          ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);<br>
          ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);<br></span>
          ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0,
      d_nnz, 0, o_nnz,&Ah);<br>
          ierr = MatCreateBAIJ(PETSC_COMM_WORLD, 3, 9, 9, 72, 72, 0,
      d_nnz, 0, o_nnz,&At);<br>
      <br>
          std::vector<double> insert(3*3*3*3, 1.0);<br>
          for(int i=0;i<8;++i)<br>
          {<br>
              int rows[3] = {i,i+1,i+3};<br>
              int cols[3] = {i,i+1,i+3};<br>
              <br>
              MatSetValuesBlocked(A, 3, rows, 3, cols, &insert[0],
      ADD_VALUES);<br>
          }<br></div></div></blockquote><div><br></div><div>I do not understand this example. All the procs (8) are filling up only the first 32 rows. Is that</div><div>what you intended? Can you tell me what output you expect?</div><div><br></div><div>  Thanks,</div><div><br></div><div>    Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div>
          MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);<br>
          MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);<br>
          MatDuplicate(A,MAT_COPY_VALUES,&Ah);<br>
          MatDuplicate(A,MAT_COPY_VALUES,&At);<br>
          MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);<br>
          MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);<br>
          MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);<br>
          MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);<br>
      <br>
          MatAXPY(Ah,1.,At,SAME_NONZERO_PATTERN);<br>
          MatAXPY(A,1.,Ah,SAME_NONZERO_PATTERN);<br>
      <br>
          MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);<br>
          MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);<br>
      <br>
      The result is, that only some values are computed correctly.<br>
      <br>
      Best and Thank you a lot<span class="HOEnZb"><font color="#888888"><br>
      <br>
      Klaus</font></span><div><div class="h5"><br>
      <br>
      <br>
      <br>
      On 03/27/2015 04:26 PM, Matthew Knepley wrote:<br>
    </div></div></div><div><div class="h5">
    <blockquote type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">On Fri, Mar 27, 2015 at 10:16 AM,
            Klaus Kaiser <span dir="ltr"><<a href="mailto:kaiser@igpm.rwth-aachen.de" target="_blank">kaiser@igpm.rwth-aachen.de</a>></span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000">
                <div>Hallo Matthew,<br>
                  <br>
                  thanks for your fast response. With "no beside the
                  nonzero structure" I meant, that I do not create a
                  different non-zero structure while I'm adding values
                  to my matrices.<br>
                  <br>
                  I also tried MAT_COPY_VALUES and the result was the
                  same.<br>
                </div>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>Please send the small example and we will figure out
              what is going wrong.</div>
            <div><br>
            </div>
            <div>  Thanks,</div>
            <div><br>
            </div>
            <div>     Matt</div>
            <div> </div>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000">
                <div> I do not get a error message but the resulting
                  matrix is wrong. Here I have a short example. The
                  first 9 rows and first 3 colomn of the matrices Ah, At
                  and Ah+At on the first processor (of 8):<br>
                  <br>
                  Ah: <br>
                  0:                   60.3553               
                  -0.249975              2.77556e-17<br>
                  1:                         0                 
                  60.3553                        0<br>
                  2:                         0                       
                  0                  60.3553<br>
                  3:                   17.6777                
                  0.374962                 0.124987<br>
                  4:                         0                 
                  17.6777                        0<br>
                  5:                         0                       
                  0                  17.6777<br>
                  6:                  -7.32233               
                  -0.124987                 0.374962<br>
                  7:                         0                
                  -7.32233                        0<br>
                  8:                         0                       
                  0                 -7.32233<br>
                  <br>
                  <br>
                  At:<br>
                  0:                         0                       
                  0                        0<br>
                  1:                      2500                       
                  0                        0<br>
                  2:              -4.54747e-13                       
                  0                        0<br>
                  3:                         0                       
                  0                        0<br>
                  4:                      1250                       
                  0                        0<br>
                  5:                      1250                       
                  0                        0<br>
                  6:                         0                       
                  0                        0<br>
                  7:                      1250                       
                  0                        0<br>
                  8:                      3750                       
                  0                        0<br>
                  <br>
                  Ah+At<br>
                  0:                   60.3553               
                  -0.249975              2.77556e-17<br>
                  1:                      2500                 
                  60.3553                        0<br>
                  2:              -4.54747e-13                       
                  0                  60.3553<br>
                  3:                   17.6777                
                  0.374962                 0.124987<br>
                  4:                         0                 
                  17.6777                        0<br>
                  5:                         0                       
                  0                  17.6777<br>
                  6:                  -7.32233               
                  -0.124987                 0.374962<br>
                  7:                         0                
                  -7.32233                        0<br>
                  8:                         0                       
                  0                 -7.32233<br>
                  <br>
                  you can see the first 3 rows of the resulting matrix
                  looks exactly like what I would expect, but the last 6
                  rows only consists of the values of Ah. When you would
                  also look on the matrix A and A+Ah+At you would also
                  see, that values where both matrices have an nonzero
                  entry are not sum correctly.<br>
                  <br>
                  Best<br>
                  <br>
                  Klaus<br>
                  <br>
                  On 03/27/2015 03:59 PM, Matthew Knepley wrote:<br>
                </div>
                <blockquote type="cite">
                  <div dir="ltr">
                    <div class="gmail_extra">
                      <div class="gmail_quote">On Fri, Mar 27, 2015 at
                        9:48 AM, Klaus Kaiser <span dir="ltr"><<a href="mailto:kaiser@igpm.rwth-aachen.de" target="_blank">kaiser@igpm.rwth-aachen.de</a>></span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hallo,<br>
                          <br>
                          I have a strange behavior in my code
                          concerning the function MatAXPY. I create 3
                          different Matrices<br>
                          <br>
                              ierr = MatCreateBAIJ(PETSC_COMM_WORLD,
                          block_size, local_size, local_size,
                          system_size, system_size, 0, d_nnz, 0,
                          o_nnz,&A);<br>
                              ierr =
                          MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);<br>
                              ierr =
                          MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);<br>
                              ierr =
                          MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);<br>
                          <br>
                              ierr = MatCreateBAIJ(PETSC_COMM_WORLD,
                          block_size, local_size, local_size,
                          system_size, system_size, 0, d_nnz, 0,
                          o_nnz,&At);<br>
                              ierr =
                          MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);<br>
                              ierr =
                          MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);<br>
                              ierr =
                          MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);<br>
                          <br>
                              ierr = MatCreateBAIJ(PETSC_COMM_WORLD,
                          block_size, local_size, local_size,
                          system_size, system_size, 0, d_nnz, 0,
                          o_nnz,&Ah);<br>
                              ierr =
                          MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);<br>
                              ierr =
                          MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);<br>
                              ierr =
                          MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);<br>
                        </blockquote>
                        <div><br>
                        </div>
                        <div>These creations are superfluous since you
                          use MatDuplicate() below.</div>
                        <div> </div>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> and want to sum these
                          three matrixes with different factors. First I
                          fill the Matrix A with some values, and
                          duplicate the structure of A to the other two
                          matrices:<br>
                          <br>
                              MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);<br>
                              MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);<br>
                             
                          MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);<br>
                             
                          MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);<br>
                              MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);<br>
                              MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);<br>
                              MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);<br>
                              MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);<br>
                        </blockquote>
                        <div> <br>
                        </div>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> After this I fill the
                          matrices At and Ah with some other values,
                          which are not beside the non zero structure (I
                          also tried with just copying the Matrix A).
                          Now after another MatAssembly</blockquote>
                        <div><br>
                        </div>
                        <div>I do not understand "no beside the nonzero
                          structure". Do you mean that the nonzero
                          structure is the same?</div>
                        <div><br>
                        </div>
                        <div>Can you first test with MAT_COPY_VALUES?</div>
                        <div> </div>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> I want to add these
                          Matrices in the form A+c*(Ah+d*At): </blockquote>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br>
                              MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);<br>
                              MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);<br>
                          <br>
                          When I run the method with mpi and one core
                          everything works fine. Starting the same
                          method with more cores, the sum of the
                          matrices fails. It seems like some values are
                          added</blockquote>
                        <div><br>
                        </div>
                        <div>Please send the full output of a failure
                          when you use MAT_COPY_VALUES on 2 procs.</div>
                        <div><br>
                        </div>
                        <div>  Thanks,</div>
                        <div><br>
                        </div>
                        <div>     Matt</div>
                        <div> </div>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> correctly and many
                          values are missed. Using
                          DIFFERENT_NONZERO_STRUCTURE leads to the right
                          behavior in the multi-core case, but is very
                          slow. I checked with a viewer if all matrices
                          have the same nonzero structure and this is
                          the case.<br>
                          <br>
                          Does anyone know why this fails, or do I have
                          made any wrong thoughts?<br>
                          <br>
                          I'm corrently working with a petsc version
                          (Petsc Release Version 3.3.0, Patch 5, Sat
                          Dec  1 15:10:41 CST 2012), I looked into the
                          changelogs up to the current version and did
                          not find any note about MatAXPY or MatAYPX.<br>
                          <br>
                          <br>
                          Best and Thanks a lot for your help<span><font color="#888888"><br>
                              <br>
                              Klaus<br>
                              <br>
                            </font></span></blockquote>
                      </div>
                      <br>
                      <br clear="all">
                      <span><font color="#888888">
                          <div><br>
                          </div>
                          -- <br>
                          <div>What most experimenters take for granted
                            before they begin their experiments is
                            infinitely more interesting than any results
                            to which their experiments lead.<br>
                            -- Norbert Wiener</div>
                        </font></span></div>
                  </div>
                </blockquote>
                <br>
              </div>
            </blockquote>
          </div>
          <br>
          <br clear="all">
          <div><br>
          </div>
          -- <br>
          <div>What most experimenters take for
            granted before they begin their experiments is infinitely
            more interesting than any results to which their experiments
            lead.<br>
            -- Norbert Wiener</div>
        </div>
      </div>
    </blockquote>
    <br>
  </div></div></div>

</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>