<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Here comes the continuation of the story:<br>
<br>
[24]PETSC ERROR: --------------------- Error Message
------------------------------------<br>
[24]PETSC ERROR: Object is in wrong state!<br>
[24]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on
argument 1 "mat" before MatScaleSystem()!<br>
[24]PETSC ERROR:
------------------------------------------------------------------------<br>
[24]PETSC ERROR: Petsc Development HG revision:
a4dae06b4d05ef5b78a9a9aa2cbc31b2b158834f HG Date: Wed Feb 08
11:29:52 2012 -0600<br>
[24]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[24]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[24]PETSC ERROR: See docs/index.html for manual pages.<br>
[24]PETSC ERROR:
------------------------------------------------------------------------<br>
[24]PETSC ERROR: /home/eprog on a openmpi-i named node226 by
agrayver Wed Feb 8 21:24:13 2012<br>
[24]PETSC ERROR: Libraries linked from
/home/lib/petsc-dev/openmpi-intel-complex-debug-f-mkl/lib<br>
[24]PETSC ERROR: Configure run at Wed Feb 8 20:51:18 2012<br>
[24]PETSC ERROR:
------------------------------------------------------------------------<br>
[24]PETSC ERROR: MatScaleSystem() line 932 in
/home/lib/petsc-dev/src/mat/interface/matrix.c<br>
[24]PETSC ERROR: PCPreSolve() line 1389 in
/home/lib/petsc-dev/src/ksp/pc/interface/precon.c<br>
[24]PETSC ERROR: KSPSolve() line 410 in
/home/lib/petsc-dev/src/ksp/ksp/interface/itfunc.c<br>
<br>
It happens within the CG solver with the system matrix which is
created like this:<br>
call
MatCreateShell(MPI_COMM_WORLD,mlocal,nlocal,N,N,PETSC_NULL_INTEGER,H,ierr);CHKERRQ(ierr)<br>
<br>
On 08.02.2012 21:28, Alexander Grayver wrote:
<blockquote cite="mid:4F32DB0B.4050002@gfz-potsdam.de" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
Hong,<br>
<br>
Everything seems to be working well so far.<br>
I would say there is nothing to acknowledge, but I don't mind...<br>
<br>
Please let me know if there will be support for mpiaij_mpidense.<br>
<br>
Thanks for this great support.<br>
<br>
On 08.02.2012 18:43, Hong Zhang wrote:
<blockquote
cite="mid:CAGCphBsuHe2wKMN1ZpjrNkv1Ev6vT3cuDSTHWNoUdmPu0Zh5yg@mail.gmail.com"
type="cite">Alexander:
<div>Recently, we add error flags in petsc-dev
requiring MatXXXSetPreallocation() for many matrix creation
routines.</div>
<div>I've updated relevant routines for your code, and add your
contributed testing code as a petsc example with acknowledge:</div>
<div>petsc-dev/src/mat/examples/tests/ex165.c</div>
<div>Let me know if you do not want your name to be acknowledge
in the example.</div>
<div><br>
</div>
<div>I tested ex165 with your A and B in sequential (np=1) and
parallel (np=2, 6):</div>
<div>mpiexec -n <np> ./ex165 -fA A.dat -fB B.dat -view_C</div>
<div><br>
</div>
<div>Checking the size of output file C.dat, I see the
sequential run and parallel run give identical bit size. </div>
<div>Please get the updated petsc-dev and test it. Let me know
if you still have problem.</div>
<div><br>
</div>
<div>It is inefficient to compute C=A^T*B via MatTranspose() and
MatMatMult().</div>
<div>I'll see if I can write a C=A^T*B for mpiaij_mpidense. </div>
<div><br>
</div>
<div>Hong</div>
<div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000"><br>
In addition, I changed my code and form A^T explicitly
(to avoid previous error) and I got another one:
<div class="im"><br>
<br>
[1]PETSC ERROR: --------------------- Error Message
------------------------------------<br>
</div>
[1]PETSC ERROR: Object is in wrong state!<br>
[1]PETSC ERROR: Must call MatXXXSetPreallocation() or
MatSetUp() on argument 1 "mat" before
MatAssemblyBegin()!<br>
[1]PETSC ERROR:
------------------------------------------------------------------------<br>
[1]PETSC ERROR: Petsc Development HG revision:
249597282bcb6a1051042a9fdfa5705679ed4f18 HG Date: Tue
Feb 07 09:44:23 2012 -0600<br>
[1]PETSC ERROR: See docs/changes/index.html for recent
updates.<br>
[1]PETSC ERROR: See docs/faq.html for hints about
trouble shooting.<br>
[1]PETSC ERROR: See docs/index.html for manual pages.<br>
[1]PETSC ERROR:
------------------------------------------------------------------------<br>
[1]PETSC ERROR: solveTest on a openmpi-i named glic1 by
agrayver Wed Feb 8 13:06:45 2012<br>
[1]PETSC ERROR: Libraries linked from
/home/lib/petsc-dev/openmpi-intel-complex-debug-f-mkl/lib<br>
[1]PETSC ERROR: Configure run at Tue Feb 7 18:19:58
2012<br>
[1]PETSC ERROR: Configure options
--with-petsc-arch=openmpi-intel-complex-debug-f-mkl
--with-fortran-interfaces=1 --download-superlu
--download-superlu_dist --download-mumps
--download-parmetis --download-ptscotch --download-metis
--with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a
--with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include
--with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a
--with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include
--with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2
--with-scalar-type=complex
--with-blas-lapack-lib="[/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_thread.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_core.a,/opt/intel/Compiler/11.1/072/lib/intel64/libiomp5.a]"
--with-precision=double --with-x=0<br>
[1]PETSC ERROR:
------------------------------------------------------------------------<br>
[1]PETSC ERROR: MatAssemblyBegin() line 4795 in
/home/lib/petsc-dev/src/mat/interface/matrix.c<br>
[1]PETSC ERROR: MatMatMultSymbolic_MPIAIJ_MPIDense()
line 638 in
/home/lib/petsc-dev/src/mat/impls/aij/mpi/mpimatmatmult.c<br>
[1]PETSC ERROR: MatMatMult_MPIAIJ_MPIDense() line 594 in
/home/lib/petsc-dev/src/mat/impls/aij/mpi/mpimatmatmult.c<br>
[1]PETSC ERROR: MatMatMult() line 8618 in
/home/lib/petsc-dev/src/mat/interface/matrix.c
<div>
<div class="h5"><br>
<br>
On 08.02.2012 17:45, Hong Zhang wrote:
<blockquote type="cite">Alexander :
<div>I can repeat the crash, and am working on it.</div>
<div>I'll let you know after the bug is fixed.</div>
<div><br>
</div>
<div>Thanks for your patience,</div>
<div>Hong<br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:
0pt 0pt 0pt 0.8ex; border-left: 1px solid
rgb(204, 204, 204); padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000"> <br>
It seems now I know why I used
MAT_REUSE_MATRIX and preallocated matrix.<br>
I removed MatCreateMPIAIJ() and use just
MatTranspose with MAT_INITIAL_MATRIX.<br>
As a consequence I get hundreds of errors
like:<br>
<br>
[30]PETSC ERROR: MatSetValues_MPIAIJ()
line 538 in
/home/mt/agrayver/lib/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c<br>
[30]PETSC ERROR: MatAssemblyEnd_MPIAIJ()
line 653 in
/home/mt/agrayver/lib/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c<br>
[30]PETSC ERROR: MatAssemblyEnd() line
4978 in
/home/mt/agrayver/lib/petsc-dev/src/mat/interface/matrix.c<br>
[30]PETSC ERROR: MatTranspose_MPIAIJ()
line 2061 in
/home/mt/agrayver/lib/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c<br>
[30]PETSC ERROR: MatTranspose() line 4397
in
/home/mt/agrayver/lib/petsc-dev/src/mat/interface/matrix.c<br>
[31]PETSC ERROR: ---------------------
Error Message
------------------------------------<br>
[31]PETSC ERROR: Argument out of range!<br>
[31]PETSC ERROR: New nonzero at
(1659,53337) caused a malloc!<br>
<br>
Ans this is the lastest petsc-dev
revision. <br>
I know there were some changes in
petsc-dev concerning this issue. <br>
Can you give me a hint how to avoid this?<br>
<br>
As for why I need C=A^T*B. This product is
used further as a system matrix for LSQR
solver. The largest dimension is on the
order of 10^6<br>
Since I form A and B myself I can, of
course, form A^T explicitly, but I thought
I would first implement everything as it
is written down on paper and then optimize
once it works (I thought it's easier way).
<div>
<div><br>
<br>
On 07.02.2012 20:44, Hong Zhang wrote:
<blockquote type="cite">Alexander,
<div>I'm curious about why do you
need parallel C=A^T*B?</div>
<div>How large your matrices are?</div>
<div><br>
</div>
<div>In petsc-dev, we
have MatTransposeMatMult() for
mpiaij and mpiaij, but not mpiaij
and mpidense.</div>
<div>We may add support
of MatTransposeMatMult_MPIAIJ_MPIDense()
if there is such need.</div>
<div><br>
</div>
<div>Hong</div>
<div><br>
<br>
<div class="gmail_quote">On Tue,
Feb 7, 2012 at 1:18 PM, <a
moz-do-not-send="true"
href="mailto:agrayver@gfz-potsdam.de"
target="_blank">agrayver@gfz-potsdam.de</a>
<span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:agrayver@gfz-potsdam.de"
target="_blank">agrayver@gfz-potsdam.de</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid
rgb(204, 204, 204);
padding-left: 1ex;">Hong,<br>
<br>
Thanks for explanation. I will
try this tomorrow. Good to
have this stuff in the help
now.<br>
<br>
And sorry for misleading you
initially.<br>
<br>
Regards,<br>
Alexander
<div>
<div><br>
<br>
<div>----- Reply message
-----<br>
From: "Hong Zhang" <<a
moz-do-not-send="true"
href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>><br>
To: "For users of the
development version of
PETSc" <<a
moz-do-not-send="true"
href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>><br>
Subject: [petsc-dev]
MatMatMult gives
different results<br>
Date: Tue, Feb 7, 2012
19:09<br>
<br>
</div>
<br>
Alexander :
<div class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin: 0pt 0pt
0pt 0.8ex;
border-left: 1px solid
rgb(204, 204, 204);
padding-left: 1ex;">
<div bgcolor="#ffffff"
text="#000000"><br>
There is something I
didn't get yet, I
hope you could
clarify it. <br>
<br>
So, when I use flag
MAT_INITIAL_MATRIX
in test program it
works fine.<br>
</div>
</blockquote>
<div>Good to know :-)</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin: 0pt 0pt
0pt 0.8ex;
border-left: 1px solid
rgb(204, 204, 204);
padding-left: 1ex;">
<div bgcolor="#ffffff"
text="#000000"> If I
put this flag in my
original program I
get dozens of
exceptions like:<br>
[42]PETSC ERROR:
Argument out of
range!<br>
[42]PETSC ERROR: New
nonzero at
(1336,153341) caused
a malloc!<br>
</div>
</blockquote>
<div>You cannot do </div>
<div> <span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">MatCreateMPIAIJ()</span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">MatTranspose(A,MAT_INITIAL_MATRIX,&AT);</span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;"><br>
</span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">MatCreateMPIAIJ()
creates AT and
preallocates
approximate
nonzeros, which does
not match exactly
the nonzeros in </span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">MatTranspose(A,MAT_INITIAL_MATRIX,&AT);</span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">MatTranspose(A,MAT_INITIAL_MATRIX,&AT)
creates matrix AT
and sets correct
nonzero pattern and
values in AT. </span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">MatTranspose()
only takes in "</span><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">MAT_INITIAL_MATRIX"
- for a new AT,</span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">and
"MAT_REUSE_MATRIX"
when AT is created
with </span><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">MatTranspose(A,MAT_INITIAL_MATRIX,&AT)</span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">and
reuse for updating
its values (not
nonzero patten).</span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;"><br>
</span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">I'm
updating petsc help
menu on </span><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">MatTranspose().
Thanks for the
report.</span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;"><br>
</span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;">Hong</span></div>
<div><span
style="border-collapse:
collapse; color:
rgb(80, 0, 80);
font-family:
arial,sans-serif;
font-size: 14px;"><br>
</span></div>
<blockquote
class="gmail_quote"
style="margin: 0pt 0pt
0pt 0.8ex;
border-left: 1px solid
rgb(204, 204, 204);
padding-left: 1ex;">
<div bgcolor="#ffffff"
text="#000000"> <br>
I changed this flag
to MAT_REUSE_MATRIX
and exceptions
disappeared, but
result is incorrect
again (same as for
MAT_IGNORE_MATRIX)
<br>
I tried test program
with
MAT_REUSE_MATRIX and
it also gives
different matrix
product.<br>
<br>
Since there is no
description of
MatReuse structure
for MatTranspose
it's a bit confusing
what to expect from
it. <br>
<div>
<div> <br>
<blockquote
type="cite">
<div
class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:
0pt 0pt 0pt
0.8ex;
border-left:
1px solid
rgb(204, 204,
204);
padding-left:
1ex;">
<div
bgcolor="#ffffff"
text="#000000">
<div>
<blockquote
type="cite">
<div>
<div
class="gmail_quote">
<div><br>
</div>
<div>Do you
mean 'Cm =
A'*B;'? </div>
<div>'Cm =
A.'*B;' gives
component-wise
matrix
product, not
matrix
product.</div>
</div>
</div>
</blockquote>
<br>
</div>
.' operator
means
non-Hermitian
transpose.
That is what I
get with
MatTranspose
(in contrast
with
MatHermitianTranspose)<br>
component-wise
matrix product
would be .*</div>
</blockquote>
<div>You are
correct.</div>
<div><br>
</div>
<div>Hong </div>
<blockquote
class="gmail_quote"
style="margin:
0pt 0pt 0pt
0.8ex;
border-left:
1px solid
rgb(204, 204,
204);
padding-left:
1ex;">
<div
bgcolor="#ffffff"
text="#000000">
<div>
<div><br>
<br>
<blockquote
type="cite">
<div>
<div
class="gmail_quote">
<div><br>
</div>
<div>Hong</div>
<div><br>
</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:
0pt 0pt 0pt
0.8ex;
border-left:
1px solid
rgb(204, 204,
204);
padding-left:
1ex;">
<div
bgcolor="#ffffff"
text="#000000">
C =
PetscBinaryRead('C.dat','complex',true);<br>
<br>
Matrix C is
different
depending on
number of
cores I use. <br>
My PETSc is:<br>
Using Petsc
Development HG
revision:
876c894d95f4fa6561d0a91310ca914592527960
HG Date: Tue
Jan 10
19:27:14 2012
+0100
<div>
<div><br>
<br>
On 06.02.2012
17:13, Hong
Zhang wrote:
<blockquote
type="cite">MatMatMult()
in petsc is
not
well-tested
for complex -
could be
buggy.
<div>Can you
send us the
matrices A and
B in petsc
binary format
for
investigation?</div>
<div><br>
</div>
<div>Hong<br>
<br>
<div
class="gmail_quote">On
Mon, Feb 6,
2012 at 5:55
AM, Alexander
Grayver <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:agrayver@gfz-potsdam.de"
target="_blank">agrayver@gfz-potsdam.de</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:
0pt 0pt 0pt
0.8ex;
border-left:
1px solid
rgb(204, 204,
204);
padding-left:
1ex;">Dear
PETSc team,<br>
<br>
I try to use:<br>
call
MatMatMult(A,B,MAT_INITIAL_MATRIX,PETSC_DEFAULT_DOUBLE_PRECISION,C,ierr);CHKERRQ(ierr)<br>
<br>
Where both A
and B are
rectangular,
but A is
sparse and B
is dense. Both
are double
complex and
distributed.<br>
The product
PETSc gives me
contains some
errors in some
part of the
matrix.<br>
I output A, B
and C then
computed
product in
matlab.<br>
<br>
Attached you
see figure
plotted as:<br>
imagesc(log10(abs(C-Cm)))<br>
<br>
Where Cm --
product
computed in
matlab.<br>
<br>
The pattern
and amplitude
vary depending
on the number
of cores I
use. This
picture is
obtained for
48 cores (I've
tried 12, 64
cores as
well).<br>
<br>
Where should I
look for
possible
explanation?<span><font
color="#888888"><br>
<br>
-- <br>
Regards,<br>
Alexander<br>
</font></span></blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<br>
</div>
</div>
<span><font
color="#888888">
<pre cols="72">--
Regards,
Alexander</pre>
</font></span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<br>
</div>
</div>
<span><font
color="#888888">
<pre cols="72">--
Regards,
Alexander</pre>
</font></span></div>
</blockquote>
</div>
<br>
</blockquote>
<br>
<br>
</div>
</div>
<span><font
color="#888888">
<pre cols="72">--
Regards,
Alexander</pre>
</font></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<br>
</div>
</div>
<span><font color="#888888">
<pre cols="72">--
Regards,
Alexander</pre>
</font></span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<br>
</div>
</div>
<span class="HOEnZb"><font color="#888888">
<pre cols="72">--
Regards,
Alexander</pre>
</font></span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Regards,
Alexander</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Regards,
Alexander</pre>
</body>
</html>