Alexander:<div>Recently, we add error flags in petsc-dev requiring MatXXXSetPreallocation() for many matrix creation routines.</div><div>I've updated relevant routines for your code, and add your contributed testing code as a petsc example with acknowledge:</div>
<div>petsc-dev/src/mat/examples/tests/ex165.c</div><div>Let me know if you do not want your name to be acknowledge in the example.</div><div><br></div><div>I tested ex165 with your A and B in sequential (np=1) and parallel (np=2, 6):</div>
<div>mpiexec -n <np> ./ex165 -fA A.dat -fB B.dat -view_C</div><div><br></div><div>Checking the size of output file C.dat, I see the sequential run and parallel run give identical bit size. </div><div>Please get the updated petsc-dev and test it. Let me know if you still have problem.</div>
<div><br></div><div>It is inefficient to compute C=A^T*B via MatTranspose() and MatMatMult().</div><div>I'll see if I can write a C=A^T*B  for mpiaij_mpidense. </div><div><br></div><div>Hong</div><div><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#ffffff" text="#000000"><br>
    In addition, I changed my code and form A^T explicitly (to avoid
    previous error) and I got another one:<div class="im"><br>
    <br>
    [1]PETSC ERROR: --------------------- Error Message
    ------------------------------------<br></div>
    [1]PETSC ERROR: Object is in wrong state!<br>
    [1]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
    argument 1 "mat" before MatAssemblyBegin()!<br>
    [1]PETSC ERROR:
    ------------------------------------------------------------------------<br>
    [1]PETSC ERROR: Petsc Development HG revision:
    249597282bcb6a1051042a9fdfa5705679ed4f18  HG Date: Tue Feb 07
    09:44:23 2012 -0600<br>
    [1]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
    [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
    [1]PETSC ERROR: See docs/index.html for manual pages.<br>
    [1]PETSC ERROR:
    ------------------------------------------------------------------------<br>
    [1]PETSC ERROR: solveTest on a openmpi-i named glic1 by agrayver Wed
    Feb  8 13:06:45 2012<br>
    [1]PETSC ERROR: Libraries linked from
    /home/lib/petsc-dev/openmpi-intel-complex-debug-f-mkl/lib<br>
    [1]PETSC ERROR: Configure run at Tue Feb  7 18:19:58 2012<br>
    [1]PETSC ERROR: Configure options
    --with-petsc-arch=openmpi-intel-complex-debug-f-mkl
    --with-fortran-interfaces=1 --download-superlu
    --download-superlu_dist --download-mumps --download-parmetis
    --download-ptscotch --download-metis
    --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a
    --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include
    --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a
    --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include
    --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2
    --with-scalar-type=complex
    --with-blas-lapack-lib="[/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_thread.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_core.a,/opt/intel/Compiler/11.1/072/lib/intel64/libiomp5.a]"
    --with-precision=double --with-x=0<br>
    [1]PETSC ERROR:
    ------------------------------------------------------------------------<br>
    [1]PETSC ERROR: MatAssemblyBegin() line 4795 in
    /home/lib/petsc-dev/src/mat/interface/matrix.c<br>
    [1]PETSC ERROR: MatMatMultSymbolic_MPIAIJ_MPIDense() line 638 in
    /home/lib/petsc-dev/src/mat/impls/aij/mpi/mpimatmatmult.c<br>
    [1]PETSC ERROR: MatMatMult_MPIAIJ_MPIDense() line 594 in
    /home/lib/petsc-dev/src/mat/impls/aij/mpi/mpimatmatmult.c<br>
    [1]PETSC ERROR: MatMatMult() line 8618 in
    /home/lib/petsc-dev/src/mat/interface/matrix.c<div><div class="h5"><br>
    <br>
    On 08.02.2012 17:45, Hong Zhang wrote:
    <blockquote type="cite">Alexander :
      <div>I can repeat the crash, and am working on it.</div>
      <div>I'll let you know after the bug is fixed.</div>
      <div><br>
      </div>
      <div>Thanks for your patience,</div>
      <div>Hong<br>
        <div class="gmail_quote">
          <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
            <div bgcolor="#ffffff" text="#000000"> <br>
              It seems now I know why I used MAT_REUSE_MATRIX and
              preallocated matrix.<br>
              I removed MatCreateMPIAIJ() and use just MatTranspose with
              MAT_INITIAL_MATRIX.<br>
              As a consequence I get hundreds of errors like:<br>
              <br>
              [30]PETSC ERROR: MatSetValues_MPIAIJ() line 538 in
              /home/mt/agrayver/lib/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c<br>
              [30]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 653 in
              /home/mt/agrayver/lib/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c<br>
              [30]PETSC ERROR: MatAssemblyEnd() line 4978 in
              /home/mt/agrayver/lib/petsc-dev/src/mat/interface/matrix.c<br>
              [30]PETSC ERROR: MatTranspose_MPIAIJ() line 2061 in
              /home/mt/agrayver/lib/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c<br>
              [30]PETSC ERROR: MatTranspose() line 4397 in
              /home/mt/agrayver/lib/petsc-dev/src/mat/interface/matrix.c<br>
              [31]PETSC ERROR: --------------------- Error Message
              ------------------------------------<br>
              [31]PETSC ERROR: Argument out of range!<br>
              [31]PETSC ERROR: New nonzero at (1659,53337) caused a
              malloc!<br>
              <br>
              Ans this is the lastest petsc-dev revision. <br>
              I know there were some changes in petsc-dev concerning
              this issue. <br>
              Can you give me a hint how to avoid this?<br>
              <br>
              As for why I need C=A^T*B. This product is used further as
              a system matrix for LSQR solver. The largest dimension is
              on the order of 10^6<br>
              Since I form A and B myself I can, of course, form A^T
              explicitly, but I thought I would first implement
              everything as it is written down on paper and then
              optimize once it works (I thought it's easier way).
              <div>
                <div><br>
                  <br>
                  On 07.02.2012 20:44, Hong Zhang wrote:
                  <blockquote type="cite">Alexander,
                    <div>I'm curious about why do you need parallel
                      C=A^T*B?</div>
                    <div>How large your matrices are?</div>
                    <div><br>
                    </div>
                    <div>In petsc-dev, we have MatTransposeMatMult() for
                      mpiaij and mpiaij, but not mpiaij and mpidense.</div>
                    <div>We may add support
                      of MatTransposeMatMult_MPIAIJ_MPIDense() if there
                      is such need.</div>
                    <div><br>
                    </div>
                    <div>Hong</div>
                    <div><br>
                      <br>
                      <div class="gmail_quote">On Tue, Feb 7, 2012 at
                        1:18 PM, <a href="mailto:agrayver@gfz-potsdam.de" target="_blank">agrayver@gfz-potsdam.de</a> <span dir="ltr"><<a href="mailto:agrayver@gfz-potsdam.de" target="_blank">agrayver@gfz-potsdam.de</a>></span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hong,<br>
                          <br>
                          Thanks for explanation. I will try this
                          tomorrow. Good to have this stuff in the help
                          now.<br>
                          <br>
                          And sorry for misleading you initially.<br>
                          <br>
                          Regards,<br>
                          Alexander
                          <div>
                            <div><br>
                              <br>
                              <div>----- Reply message -----<br>
                                From: "Hong Zhang" <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>><br>
                                To: "For users of the development
                                version of PETSc" <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>><br>
                                Subject: [petsc-dev] MatMatMult gives
                                different results<br>
                                Date: Tue, Feb 7, 2012 19:09<br>
                                <br>
                              </div>
                              <br>
                              Alexander :
                              <div class="gmail_quote">
                                <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                  <div bgcolor="#ffffff" text="#000000"><br>
                                    There is something I didn't get yet,
                                    I hope you could clarify it. <br>
                                    <br>
                                    So, when I use flag
                                    MAT_INITIAL_MATRIX in test program
                                    it works fine.<br>
                                  </div>
                                </blockquote>
                                <div>Good to know :-)</div>
                                <div> </div>
                                <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                  <div bgcolor="#ffffff" text="#000000">
                                    If I put this flag in my original
                                    program I get dozens of exceptions
                                    like:<br>
                                    [42]PETSC ERROR: Argument out of
                                    range!<br>
                                    [42]PETSC ERROR: New nonzero at
                                    (1336,153341) caused a malloc!<br>
                                  </div>
                                </blockquote>
                                <div>You cannot do </div>
                                <div> <span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatCreateMPIAIJ()</span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatTranspose(A,MAT_INITIAL_MATRIX,&AT);</span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px"><br>
                                  </span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatCreateMPIAIJ()
                                    creates AT and preallocates
                                    approximate nonzeros, which does not
                                    match exactly the nonzeros in </span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatTranspose(A,MAT_INITIAL_MATRIX,&AT);</span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatTranspose(A,MAT_INITIAL_MATRIX,&AT)

                                    creates matrix AT and sets correct
                                    nonzero pattern and values in AT. </span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatTranspose()
                                    only takes in "</span><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MAT_INITIAL_MATRIX"
                                    - for a new AT,</span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">and
                                    "MAT_REUSE_MATRIX" when AT is
                                    created with </span><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatTranspose(A,MAT_INITIAL_MATRIX,&AT)</span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">and reuse for
                                    updating its values (not nonzero
                                    patten).</span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px"><br>
                                  </span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">I'm updating petsc
                                    help menu on </span><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">MatTranspose().
                                    Thanks for the report.</span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px"><br>
                                  </span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">Hong</span></div>
                                <div><span style="border-collapse:collapse;color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px"><br>
                                  </span></div>
                                <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                  <div bgcolor="#ffffff" text="#000000">
                                    <br>
                                    I changed this flag to
                                    MAT_REUSE_MATRIX and exceptions
                                    disappeared, but result is incorrect
                                    again (same as for
                                    MAT_IGNORE_MATRIX)  <br>
                                    I tried test program with
                                    MAT_REUSE_MATRIX and it also gives
                                    different matrix product.<br>
                                    <br>
                                    Since there is no description of
                                    MatReuse structure for MatTranspose
                                    it's a bit confusing what to expect
                                    from it. <br>
                                    <div>
                                      <div> <br>
                                        <blockquote type="cite">
                                          <div class="gmail_quote">
                                            <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                              <div bgcolor="#ffffff" text="#000000">
                                                <div>
                                                  <blockquote type="cite">
                                                    <div>
                                                      <div class="gmail_quote">
                                                        <div><br>
                                                        </div>
                                                        <div>Do you mean
                                                          'Cm = A'*B;'? </div>
                                                        <div>'Cm =
                                                          A.'*B;' gives
                                                          component-wise
                                                          matrix
                                                          product, not
                                                          matrix
                                                          product.</div>
                                                      </div>
                                                    </div>
                                                  </blockquote>
                                                  <br>
                                                </div>
                                                .' operator means
                                                non-Hermitian transpose.
                                                That is what I get with
                                                MatTranspose (in
                                                contrast with
                                                MatHermitianTranspose)<br>
                                                component-wise matrix
                                                product would be .*</div>
                                            </blockquote>
                                            <div>You are correct.</div>
                                            <div><br>
                                            </div>
                                            <div>Hong </div>
                                            <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                              <div bgcolor="#ffffff" text="#000000">
                                                <div>
                                                  <div><br>
                                                    <br>
                                                    <blockquote type="cite">
                                                      <div>
                                                        <div class="gmail_quote">
                                                          <div><br>
                                                          </div>
                                                          <div>Hong</div>
                                                          <div><br>
                                                          </div>
                                                          <div> </div>
                                                          <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#ffffff" text="#000000">
                                                          C =
                                                          PetscBinaryRead('C.dat','complex',true);<br>
                                                          <br>
                                                          Matrix C is
                                                          different
                                                          depending on
                                                          number of
                                                          cores I use. <br>
                                                          My PETSc is:<br>
                                                          Using Petsc
                                                          Development HG
                                                          revision:
                                                          876c894d95f4fa6561d0a91310ca914592527960 
                                                          HG Date: Tue
                                                          Jan 10
                                                          19:27:14 2012
                                                          +0100
                                                          <div>
                                                          <div><br>
                                                          <br>
                                                          On 06.02.2012
                                                          17:13, Hong
                                                          Zhang wrote:
                                                          <blockquote type="cite">MatMatMult()

                                                          in petsc is
                                                          not
                                                          well-tested
                                                          for complex -
                                                          could be
                                                          buggy.
                                                          <div>Can you
                                                          send us the
                                                          matrices A and
                                                          B in petsc
                                                          binary format
                                                          for
                                                          investigation?</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Hong<br>
                                                          <br>
                                                          <div class="gmail_quote">On

                                                          Mon, Feb 6,
                                                          2012 at 5:55
                                                          AM, Alexander
                                                          Grayver <span dir="ltr"><<a href="mailto:agrayver@gfz-potsdam.de" target="_blank">agrayver@gfz-potsdam.de</a>></span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear
                                                          PETSc team,<br>
                                                          <br>
                                                          I try to use:<br>
                                                          call
MatMatMult(A,B,MAT_INITIAL_MATRIX,PETSC_DEFAULT_DOUBLE_PRECISION,C,ierr);CHKERRQ(ierr)<br>
                                                          <br>
                                                          Where both A
                                                          and B are
                                                          rectangular,
                                                          but A is
                                                          sparse and B
                                                          is dense. Both
                                                          are double
                                                          complex and
                                                          distributed.<br>
                                                          The product
                                                          PETSc gives me
                                                          contains some
                                                          errors in some
                                                          part of the
                                                          matrix.<br>
                                                          I output A, B
                                                          and C then
                                                          computed
                                                          product in
                                                          matlab.<br>
                                                          <br>
                                                          Attached you
                                                          see figure
                                                          plotted as:<br>
imagesc(log10(abs(C-Cm)))<br>
                                                          <br>
                                                          Where Cm --
                                                          product
                                                          computed in
                                                          matlab.<br>
                                                          <br>
                                                          The pattern
                                                          and amplitude
                                                          vary depending
                                                          on the number
                                                          of cores I
                                                          use. This
                                                          picture is
                                                          obtained for
                                                          48 cores (I've
                                                          tried 12, 64
                                                          cores as
                                                          well).<br>
                                                          <br>
                                                          Where should I
                                                          look for
                                                          possible
                                                          explanation?<span><font color="#888888"><br>
                                                          <br>
                                                          -- <br>
                                                          Regards,<br>
                                                          Alexander<br>
                                                          </font></span></blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          <span><font color="#888888">
                                                          <pre cols="72">-- 
Regards,
Alexander</pre>
                                                          </font></span></div>
                                                          </blockquote>
                                                        </div>
                                                        <br>
                                                      </div>
                                                    </blockquote>
                                                    <br>
                                                    <br>
                                                  </div>
                                                </div>
                                                <span><font color="#888888">
                                                    <pre cols="72">-- 
Regards,
Alexander</pre>
                                                  </font></span></div>
                                            </blockquote>
                                          </div>
                                          <br>
                                        </blockquote>
                                        <br>
                                        <br>
                                      </div>
                                    </div>
                                    <span><font color="#888888">
                                        <pre cols="72">-- 
Regards,
Alexander</pre>
                                      </font></span></div>
                                </blockquote>
                              </div>
                              <br>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                      <br>
                    </div>
                  </blockquote>
                  <br>
                  <br>
                </div>
              </div>
              <span><font color="#888888">
                  <pre cols="72">-- 
Regards,
Alexander</pre>
                </font></span></div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
    <br>
    </div></div><span class="HOEnZb"><font color="#888888"><pre cols="72">-- 
Regards,
Alexander</pre>
  </font></span></div>

</blockquote></div><br></div>