Alexander :<div>I can repeat the crash, and am working on it.</div><div>I'll let you know after the bug is fixed.</div><div><br></div><div>Thanks for your patience,</div><div>Hong<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#ffffff" text="#000000">
    <br>
    It seems now I know why I used MAT_REUSE_MATRIX and preallocated
    matrix.<br>
    I removed MatCreateMPIAIJ() and use just MatTranspose with
    MAT_INITIAL_MATRIX.<br>
    As a consequence I get hundreds of errors like:<br>
    <br>
    [30]PETSC ERROR: MatSetValues_MPIAIJ() line 538 in
    /home/mt/agrayver/lib/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c<br>
    [30]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 653 in
    /home/mt/agrayver/lib/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c<br>
    [30]PETSC ERROR: MatAssemblyEnd() line 4978 in
    /home/mt/agrayver/lib/petsc-dev/src/mat/interface/matrix.c<br>
    [30]PETSC ERROR: MatTranspose_MPIAIJ() line 2061 in
    /home/mt/agrayver/lib/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c<br>
    [30]PETSC ERROR: MatTranspose() line 4397 in
    /home/mt/agrayver/lib/petsc-dev/src/mat/interface/matrix.c<br>
    [31]PETSC ERROR: --------------------- Error Message
    ------------------------------------<br>
    [31]PETSC ERROR: Argument out of range!<br>
    [31]PETSC ERROR: New nonzero at (1659,53337) caused a malloc!<br>
    <br>
    Ans this is the lastest petsc-dev revision. <br>
    I know there were some changes in petsc-dev concerning this issue. <br>
    Can you give me a hint how to avoid this?<br>
    <br>
    As for why I need C=A^T*B. This product is used further as a system
    matrix for LSQR solver. The largest dimension is on the order of
    10^6<br>
    Since I form A and B myself I can, of course, form A^T explicitly,
    but I thought I would first implement everything as it is written
    down on paper and then optimize once it works (I thought it's easier
    way).<div><div class="h5"><br>
    <br>
    On 07.02.2012 20:44, Hong Zhang wrote:
    <blockquote type="cite">Alexander,
      <div>I'm curious about why do you need parallel C=A^T*B?</div>
      <div>How large your matrices are?</div>
      <div><br>
      </div>
      <div>In petsc-dev, we have MatTransposeMatMult() for mpiaij and
        mpiaij, but not mpiaij and mpidense.</div>
      <div>We may add support of MatTransposeMatMult_MPIAIJ_MPIDense()
        if there is such need.</div>
      <div><br>
      </div>
      <div>Hong</div>
      <div><br>
        <br>
        <div class="gmail_quote">On Tue, Feb 7, 2012 at 1:18 PM, <a href="mailto:agrayver@gfz-potsdam.de" target="_blank">agrayver@gfz-potsdam.de</a>
          <span dir="ltr"><<a href="mailto:agrayver@gfz-potsdam.de" target="_blank">agrayver@gfz-potsdam.de</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hong,<br>
            <br>
            Thanks for explanation. I will try this tomorrow. Good to
            have this stuff in the help now.<br>
            <br>
            And sorry for misleading you initially.<br>
            <br>
            Regards,<br>
            Alexander
            <div>
              <div><br>
                <br>
                <div>----- Reply message -----<br>
                  From: "Hong Zhang" <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>><br>
                  To: "For users of the development version of PETSc"
                  <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>><br>
                  Subject: [petsc-dev] MatMatMult gives different
                  results<br>
                  Date: Tue, Feb 7, 2012 19:09<br>
                  <br>
                </div>
                <br>
                Alexander :
                <div class="gmail_quote">
                  <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                    <div bgcolor="#ffffff" text="#000000"><br>
                      There is something I didn't get yet, I hope you
                      could clarify it. <br>
                      <br>
                      So, when I use flag MAT_INITIAL_MATRIX in test
                      program it works fine.<br>
                    </div>
                  </blockquote>
                  <div>Good to know :-)</div>
                  <div> </div>
                  <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                    <div bgcolor="#ffffff" text="#000000"> If I put this
                      flag in my original program I get dozens of
                      exceptions like:<br>
                      [42]PETSC ERROR: Argument out of range!<br>
                      [42]PETSC ERROR: New nonzero at (1336,153341)
                      caused a malloc!<br>
                    </div>
                  </blockquote>
                  <div>You cannot do </div>
                  <div> <span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatCreateMPIAIJ()</span></div>
                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatTranspose(A,MAT_INITIAL_MATRIX,&AT);</span></div>
                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px"><br>
                    </span></div>
                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatCreateMPIAIJ() creates AT and
                      preallocates approximate nonzeros, which does not
                      match exactly the nonzeros in </span></div>
                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatTranspose(A,MAT_INITIAL_MATRIX,&AT);</span></div>
                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatTranspose(A,MAT_INITIAL_MATRIX,&AT)
                      creates matrix AT and sets correct nonzero pattern
                      and values in AT. </span></div>
                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatTranspose() only takes in "</span><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MAT_INITIAL_MATRIX" - for a new AT,</span></div>

                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">and "MAT_REUSE_MATRIX" when AT
                      is created with </span><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatTranspose(A,MAT_INITIAL_MATRIX,&AT)</span></div>
                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">and reuse for updating its
                      values (not nonzero patten).</span></div>
                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px"><br>
                    </span></div>
                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">I'm updating petsc help menu on </span><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatTranspose(). Thanks for the report.</span></div>

                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px"><br>
                    </span></div>
                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">Hong</span></div>
                  <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px"><br>
                    </span></div>
                  <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                    <div bgcolor="#ffffff" text="#000000"> <br>
                      I changed this flag to MAT_REUSE_MATRIX and
                      exceptions disappeared, but result is incorrect
                      again (same as for MAT_IGNORE_MATRIX)  <br>
                      I tried test program with MAT_REUSE_MATRIX and it
                      also gives different matrix product.<br>
                      <br>
                      Since there is no description of MatReuse
                      structure for MatTranspose it's a bit confusing
                      what to expect from it. <br>
                      <div>
                        <div> <br>
                          <blockquote type="cite">
                            <div class="gmail_quote">
                              <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                <div bgcolor="#ffffff" text="#000000">
                                  <div>
                                    <blockquote type="cite">
                                      <div>
                                        <div class="gmail_quote">
                                          <div><br>
                                          </div>
                                          <div>Do you mean 'Cm =
                                            A'*B;'? </div>
                                          <div>'Cm = A.'*B;' gives
                                            component-wise matrix
                                            product, not matrix product.</div>
                                        </div>
                                      </div>
                                    </blockquote>
                                    <br>
                                  </div>
                                  .' operator means non-Hermitian
                                  transpose. That is what I get with
                                  MatTranspose (in contrast with
                                  MatHermitianTranspose)<br>
                                  component-wise matrix product would be
                                  .*</div>
                              </blockquote>
                              <div>You are correct.</div>
                              <div><br>
                              </div>
                              <div>Hong </div>
                              <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                <div bgcolor="#ffffff" text="#000000">
                                  <div>
                                    <div><br>
                                      <br>
                                      <blockquote type="cite">
                                        <div>
                                          <div class="gmail_quote">
                                            <div><br>
                                            </div>
                                            <div>Hong</div>
                                            <div><br>
                                            </div>
                                            <div> </div>
                                            <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                              <div bgcolor="#ffffff" text="#000000"> C =
                                                PetscBinaryRead('C.dat','complex',true);<br>
                                                <br>
                                                Matrix C is different
                                                depending on number of
                                                cores I use. <br>
                                                My PETSc is:<br>
                                                Using Petsc Development
                                                HG revision:
                                                876c894d95f4fa6561d0a91310ca914592527960 
                                                HG Date: Tue Jan 10
                                                19:27:14 2012 +0100
                                                <div>
                                                  <div><br>
                                                    <br>
                                                    On 06.02.2012 17:13,
                                                    Hong Zhang wrote:
                                                    <blockquote type="cite">MatMatMult()
                                                      in petsc is not
                                                      well-tested for
                                                      complex - could be
                                                      buggy.
                                                      <div>Can you send
                                                        us the matrices
                                                        A and B in petsc
                                                        binary format
                                                        for
                                                        investigation?</div>
                                                      <div><br>
                                                      </div>
                                                      <div>Hong<br>
                                                        <br>
                                                        <div class="gmail_quote">On
                                                          Mon, Feb 6,
                                                          2012 at 5:55
                                                          AM, Alexander
                                                          Grayver <span dir="ltr"><<a href="mailto:agrayver@gfz-potsdam.de" target="_blank">agrayver@gfz-potsdam.de</a>></span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear
                                                          PETSc team,<br>
                                                          <br>
                                                          I try to use:<br>
                                                          call
MatMatMult(A,B,MAT_INITIAL_MATRIX,PETSC_DEFAULT_DOUBLE_PRECISION,C,ierr);CHKERRQ(ierr)<br>
                                                          <br>
                                                          Where both A
                                                          and B are
                                                          rectangular,
                                                          but A is
                                                          sparse and B
                                                          is dense. Both
                                                          are double
                                                          complex and
                                                          distributed.<br>
                                                          The product
                                                          PETSc gives me
                                                          contains some
                                                          errors in some
                                                          part of the
                                                          matrix.<br>
                                                          I output A, B
                                                          and C then
                                                          computed
                                                          product in
                                                          matlab.<br>
                                                          <br>
                                                          Attached you
                                                          see figure
                                                          plotted as:<br>
imagesc(log10(abs(C-Cm)))<br>
                                                          <br>
                                                          Where Cm --
                                                          product
                                                          computed in
                                                          matlab.<br>
                                                          <br>
                                                          The pattern
                                                          and amplitude
                                                          vary depending
                                                          on the number
                                                          of cores I
                                                          use. This
                                                          picture is
                                                          obtained for
                                                          48 cores (I've
                                                          tried 12, 64
                                                          cores as
                                                          well).<br>
                                                          <br>
                                                          Where should I
                                                          look for
                                                          possible
                                                          explanation?<span><font color="#888888"><br>
                                                          <br>
                                                          -- <br>
                                                          Regards,<br>
                                                          Alexander<br>
                                                          </font></span></blockquote>
                                                        </div>
                                                        <br>
                                                      </div>
                                                    </blockquote>
                                                    <br>
                                                    <br>
                                                  </div>
                                                </div>
                                                <span><font color="#888888">
                                                    <pre cols="72">-- 
Regards,
Alexander</pre>
                                                  </font></span></div>
                                            </blockquote>
                                          </div>
                                          <br>
                                        </div>
                                      </blockquote>
                                      <br>
                                      <br>
                                    </div>
                                  </div>
                                  <span><font color="#888888">
                                      <pre cols="72">-- 
Regards,
Alexander</pre>
                                    </font></span></div>
                              </blockquote>
                            </div>
                            <br>
                          </blockquote>
                          <br>
                          <br>
                        </div>
                      </div>
                      <span><font color="#888888">
                          <pre cols="72">-- 
Regards,
Alexander</pre>
                        </font></span></div>
                  </blockquote>
                </div>
                <br>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
    <br>
    </div></div><span class="HOEnZb"><font color="#888888"><pre cols="72">-- 
Regards,
Alexander</pre>
  </font></span></div>

</blockquote></div><br></div>