<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>Hei,</p>
<p>I think I found a solution, but no explanation for it (yet). In
my CMakeLists-file I was linking the following libraries:</p>
<p> ${EIGEN3_LIBRARIES}<br>
${GSL_LIBRARIES}<br>
${FFTW_LIBRARIES}<br>
/opt/fftw3/lib64/libfftw3_mpi.so<br>
gfortran<br>
/opt/intel/mkl/lib/intel64/libmkl_rt.so<br>
${PETSC_LIBRARY_DIRS}/libpetsc.so<br>
Boost::filesystem<br>
Boost::mpi<br>
Boost::program_options<br>
Boost::serialization<br>
${X11_LIBRARIES}<br>
OpenMP::OpenMP_CXX</p>
<p>with <br>
</p>
<p> ${FFTW_LIBRARIES}</p>
<p>containing</p>
<p> libfftw3.so<br>
libfftw3_omp.so</p>
<p>Now, apparently the issue was a clash between the
OpenMP-libraries and mkl_rt. When removing mkl_rt, the program
works as expected, and similarly for when removing
OpenMP::OpenMP_CXX and libfftw3_omp.so. As long as both options
are present in the link list, I will obtain wrong results.
Therefore, I'll now take a deeper look at what causes that
behavior. Nevertheless, for the moment it looks as if I could
solve the problem. <br>
</p>
<p>Thank you very much for your help!</p>
<p>Regards,</p>
<p>Roland<br>
</p>
<div class="moz-cite-prefix">Am 06.01.21 um 19:44 schrieb Barry
Smith:<br>
</div>
<blockquote type="cite"
cite="mid:F3186FEE-5413-4670-89AD-5E411B538B8A@petsc.dev">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<br class="">
<div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">$
./main -start_in_debugger noxterm</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">PETSC:
Attaching lldb to ./main of pid 17914 on
Barry-Smiths-MacBook-Pro.local</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">(lldb)
process attach --pid 17914</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">warning:
(x86_64)
/Users/barrysmith/soft/clang-ifort/lib/libmpifort.12.dylib
empty dSYM file detected, dSYM was created with an
executable with no debug info.</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">warning:
(x86_64)
/Users/barrysmith/soft/clang-ifort/lib/libmpi.12.dylib empty
dSYM file detected, dSYM was created with an executable with
no debug info.</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">warning:
(x86_64)
/Users/barrysmith/soft/clang-ifort/lib/libpmpi.12.dylib
empty dSYM file detected, dSYM was created with an
executable with no debug info.</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">Process
17914 stopped</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">*
thread #1, queue = </span><span
style="font-variant-ligatures: no-common-ligatures; color:
#2fb41d" class="">'com.apple.main-thread'</span><span
style="font-variant-ligatures: no-common-ligatures" class="">,
stop reason = </span><span style="font-variant-ligatures:
no-common-ligatures; color: #b42419" class="">signal SIGSTOP</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">
frame #0: </span><span style="font-variant-ligatures:
no-common-ligatures; color: #9fa01c" class="">0x00007fff733cb756</span><span
style="font-variant-ligatures: no-common-ligatures" class="">
libsystem_kernel.dylib`__semwait_signal + 10</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">libsystem_kernel.dylib`__semwait_signal:</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures; color:
#9fa01c" class="">-> </span><span
style="font-variant-ligatures: no-common-ligatures" class="">
0x7fff733cb756 <+10>: jae 0x7fff733cb760
; <+20></span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures; color:
#9fa01c" class=""> </span><span
style="font-variant-ligatures: no-common-ligatures" class="">
0x7fff733cb758 <+12>: movq %rax, %rdi</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures; color:
#9fa01c" class=""> </span><span
style="font-variant-ligatures: no-common-ligatures" class="">
0x7fff733cb75b <+15>: jmp 0x7fff733ca22d
; cerror</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures; color:
#9fa01c" class=""> </span><span
style="font-variant-ligatures: no-common-ligatures" class="">
0x7fff733cb760 <+20>: retq </span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">Target
0: (main) stopped.</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo; min-height: 16px;"
class=""><span style="font-variant-ligatures:
no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">Executable
module set to
"/Users/barrysmith/Src/petsc/src/ksp/ksp/tutorials/main".</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">Architecture
set to: x86_64h-apple-macosx-.</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures; color:
#999999" class="">(lldb) </span><span
style="font-variant-ligatures: no-common-ligatures" class="">b
MatScale</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures" class="">Breakpoint
1: where = libpetsc.3.014.dylib`MatScale + 48 at
matrix.c:5281:3, address = 0x0000000102f62090</span></div>
<div style="margin: 0px; font-stretch: normal; font-size: 14px;
line-height: normal; font-family: Menlo; color: rgb(153, 153,
153);" class=""><span style="font-variant-ligatures:
no-common-ligatures" class="">(lldb) </span><span
style="font-variant-ligatures: no-common-ligatures; color:
#000000" class="">c</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures; color: #000000" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures; color: #000000" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures; color: #000000" class="">
<div style="margin: 0px; font-stretch: normal; font-size:
14px; line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class="">1.0230000000000000e+03 </span></div>
<div style="margin: 0px; font-stretch: normal; font-size:
14px; line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class="">Process 17914 stopped</span></div>
<div style="margin: 0px; font-stretch: normal; font-size:
14px; line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class="">* thread #1, queue = </span><span
style="font-variant-ligatures: no-common-ligatures;
color: #2fb41d" class="">'com.apple.main-thread'</span><span
style="font-variant-ligatures: no-common-ligatures"
class="">, stop reason = </span><span
style="font-variant-ligatures: no-common-ligatures;
color: #b42419" class="">breakpoint 1.1</span></div>
<div style="margin: 0px; font-stretch: normal; font-size:
14px; line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class=""> frame #0: </span><span
style="font-variant-ligatures: no-common-ligatures;
color: #9fa01c" class="">0x0000000102f62090</span><span
style="font-variant-ligatures: no-common-ligatures"
class="">
libpetsc.3.014.dylib`MatScale(mat=0x00007fa5040c9870,
a=0.01 + 0i) at </span><span
style="font-variant-ligatures: no-common-ligatures;
color: #2eaebb" class="">matrix.c</span><span
style="font-variant-ligatures: no-common-ligatures"
class="">:</span><span style="font-variant-ligatures:
no-common-ligatures; color: #9fa01c" class="">5281</span><span
style="font-variant-ligatures: no-common-ligatures"
class="">:</span><span style="font-variant-ligatures:
no-common-ligatures; color: #9fa01c" class="">3</span></div>
<div style="margin: 0px; font-stretch: normal; font-size:
14px; line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures;
color: #9fa01c" class=""> </span><span
style="font-variant-ligatures: no-common-ligatures"
class=""> 5278<span class="Apple-tab-span" style="white-space:pre"> </span>{</span></div>
<div style="margin: 0px; font-stretch: normal; font-size:
14px; line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures;
color: #9fa01c" class=""> </span><span
style="font-variant-ligatures: no-common-ligatures"
class=""> 5279<span class="Apple-tab-span" style="white-space:pre"> </span>
PetscErrorCode ierr;</span></div>
<div style="margin: 0px; font-stretch: normal; font-size:
14px; line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures;
color: #9fa01c" class=""> </span><span
style="font-variant-ligatures: no-common-ligatures"
class=""> 5280<span class="Apple-tab-span" style="white-space:pre"> </span></span></div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; font-size:
14px; line-height: normal; font-family: Menlo;" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class="">Target 0: (main) stopped.</span></div>
<div style="margin: 0px; font-stretch: normal; font-size:
14px; line-height: normal; font-family: Menlo; color:
rgb(153, 153, 153);" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class="">(lldb) </span><span
style="font-variant-ligatures: no-common-ligatures;
color: #000000" class="">c</span></div>
</div>
<div style="margin: 0px; font-stretch: normal; font-size:
14px; line-height: normal; font-family: Menlo; color:
rgb(153, 153, 153);" class=""><span
style="font-variant-ligatures: no-common-ligatures;
color: #000000" class=""><br class="">
</span></div>
<div style="margin: 0px; font-stretch: normal; font-size:
14px; line-height: normal; font-family: Menlo; color:
rgb(153, 153, 153);" class=""><span
style="font-variant-ligatures: no-common-ligatures;
color: #000000" class="">
<div style="margin: 0px; font-stretch: normal;
line-height: normal;" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class="">0001e+01 1.0230000000000000e+01 </span></div>
<div style="margin: 0px; font-stretch: normal;
line-height: normal;" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class="">[ 0]32 bytes PetscPushErrorHandler() line
161 in
/Users/barrysmith/Src/petsc/src/sys/error/err.c</span></div>
<div style="margin: 0px; font-stretch: normal;
line-height: normal;" class=""><span
style="font-variant-ligatures: no-common-ligatures"
class="">Process 17914 exited with status = 0
(0x00000000) </span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class="">I see no issue. Code
is, of course, built with complex.</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class="">Barry</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class=""><br class="">
</span></div>
</span></div>
</span></div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 6, 2021, at 11:31 AM, Roland Richter <<a
href="mailto:roland.richter@ntnu.no" class=""
moz-do-not-send="true">roland.richter@ntnu.no</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<p class="">Hei,</p>
<p class="">I removed all dependencies to armadillo and
other not directly necessary packages, and attached both
CMakeLists.txt and main-file. Even though I am only
having PETSc as main dependence I still have the same
issues. For a scaling factor of 0.1 and a matrix size of
[1024, 1024] it works fine, for a scaling factor of 0.01
on the same matrix the apparent scaling factor is
suddenly 1e-8.</p>
<p class="">Thank you for your help!</p>
<p class="">Regards,</p>
<p class="">Roland<br class="">
</p>
<div class="moz-cite-prefix">Am 06.01.21 um 17:36 schrieb
Matthew Knepley:<br class="">
</div>
<blockquote type="cite"
cite="mid:CAMYG4G=oHnxt1PhOMKM3SJfxooOesu00M2aX2yjVe-rpBi-pEw@mail.gmail.com"
class="">
<div dir="ltr" class="">
<div dir="ltr" class="">On Wed, Jan 6, 2021 at 11:05
AM Roland Richter <<a
href="mailto:roland.richter@ntnu.no"
moz-do-not-send="true" class="">roland.richter@ntnu.no</a>>
wrote:<br class="">
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div class="">
<p class="">Hei,</p>
<p class="">I ran the program in both versions
using "valgrind --tool=memcheck
--leak-check=full --show-leak-kinds=all
<binary> -malloc_debug". I got</p>
<p class="">==3059== LEAK SUMMARY:<br class="">
==3059== definitely lost: 12,916 bytes in
32 blocks<br class="">
==3059== indirectly lost: 2,415 bytes in 2
blocks<br class="">
==3059== possibly lost: 0 bytes in 0
blocks<br class="">
==3059== still reachable: 103,511 bytes in
123 blocks<br class="">
==3059== suppressed: 0 bytes in 0
blocks<br class="">
</p>
<p class="">but none of the leaks is related to
the scaling-function itself.</p>
<p class="">Did I miss something here?</p>
</div>
</blockquote>
<div class="">Here is my analysis. It is certainly
the case that MatScale() does not mysteriously
scale by other numbers.</div>
<div class="">It is used all over the place in
tests, and in the code. Your test requires another
package. Thus, it seems</div>
<div class="">reasonable to guess that a bad
interaction with that package (memory overwrite,
conflicting layout or format, etc.)</div>
<div class="">is responsible for the behavior you
see.</div>
<div class=""><br class="">
</div>
<div class=""> Thanks,</div>
<div class=""><br class="">
</div>
<div class=""> Matt </div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div class="">
<p class="">Thanks!<br class="">
</p>
<div class="">Am 06.01.21 um 15:26 schrieb
Matthew Knepley:<br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div dir="ltr" class="">On Wed, Jan 6, 2021
at 2:41 AM Roland Richter <<a
href="mailto:roland.richter@ntnu.no"
target="_blank" moz-do-not-send="true"
class="">roland.richter@ntnu.no</a>>
wrote:<br class="">
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div class="">
<p class="">Hei,</p>
<p class="">I added one additional
function to the code:</p>
<p class=""><i class="">void
test_scaling_petsc_pointer(const
Mat &in_mat,</i><i class=""><br
class="">
</i><i class="">
Mat &out_mat,</i><i
class=""><br class="">
</i><i class="">
const PetscScalar
&scaling_factor) {</i><i
class=""><br class="">
</i><i class=""> MatCopy (in_mat,
out_mat, SAME_NONZERO_PATTERN);</i><i
class=""><br class="">
</i><i class=""> PetscScalar
*mat_ptr;</i><i class=""><br
class="">
</i><i class=""> MatDenseGetArray
(out_mat, &mat_ptr);</i><i
class=""><br class="">
</i><i class=""> PetscInt r_0,
r_1;</i><i class=""><br class="">
</i><i class=""> MatGetLocalSize
(out_mat, &r_0, &r_1);</i><i
class=""><br class="">
</i><i class=""> for(int i = 0; i
< r_0 * r_1; ++i)</i><i
class=""><br class="">
</i><i class=""> *(mat_ptr +
i) = (*(mat_ptr + i) *
scaling_factor);</i><i class=""><br
class="">
</i><i class=""><br class="">
</i><i class=""> MatAssemblyBegin
(out_mat, MAT_FINAL_ASSEMBLY);</i><i
class=""><br class="">
</i><i class=""> MatAssemblyEnd
(out_mat, MAT_FINAL_ASSEMBLY);</i><i
class=""><br class="">
</i><i class="">}</i></p>
<p class="">When replacing test
function <i class="">test_scaling_petsc()</i>
with <i class="">test_scaling_petsc_pointer()</i>
everything works as it should, but I
do not understand why.</p>
<p class="">Do you have any
suggestions?</p>
</div>
</blockquote>
<div class="">The easiest explanation is
that you have a memory overwrite in the
code somewhere. Barry's suggestion to
use</div>
<div class="">valgrind is good.</div>
<div class=""><br class="">
</div>
<div class=""> Matt </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div class="">
<p class="">Thanks!<br class="">
</p>
<p class=""><br class="">
</p>
<div class="">Am 05.01.21 um 15:24
schrieb Roland Richter:<br class="">
</div>
<blockquote type="cite" class="">
<p class="">Hei,</p>
<p class="">the code I attached to
the original mail should work out
of the box, but requires armadillo
and PETSc to compile/run.
Armadillo stores the data in
column-major order, and therefore
I am transposing the matrices
before and after transferring
using .st().</p>
<p class="">Thank you for your help!</p>
<p class="">Regards,</p>
<p class="">Roland<br class="">
</p>
<div class="">Am 05.01.21 um 15:21
schrieb Matthew Knepley:<br
class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div dir="ltr" class="">On Tue,
Jan 5, 2021 at 7:57 AM Roland
Richter <<a
href="mailto:roland.richter@ntnu.no"
target="_blank"
moz-do-not-send="true"
class="">roland.richter@ntnu.no</a>>
wrote:<br class="">
</div>
<div class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Hei,<br class="">
<br class="">
I would like to scale a
given matrix with a fixed
scalar value, and<br
class="">
therefore would like to use
MatScale(). Nevertheless, I
observed an<br class="">
interesting behavior
depending on the size of the
matrix, and currently<br
class="">
I am not sure why.<br
class="">
<br class="">
When running the attached
code, I intend to divide all
elements in the<br class="">
matrix by a constant factor
of 10. If I have three or
fewer rows and<br class="">
1024 columns, I get the
expected result. If I have
four or more rows<br
class="">
(with the same number of
columns), suddenly my
scaling factor seems to<br
class="">
be 0.01 instead of 0.1 for
the PETSc-matrix. The
armadillo-based matrix<br
class="">
still behaves as expected.<br
class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">1) It looks like
you assume the storage in
your armadillo matrix is row
major. I would be surprised
if this was true.</div>
<div class=""><br class="">
</div>
<div class="">2) I think it is
unlikely that there is a
problem with MatScale, so I
would guess either you have
a memory overwrite</div>
<div class="">or are
misinterpreting your output.
If you send something I can
run, I will figure out which
it is.</div>
<div class=""><br class="">
</div>
<div class=""> Thanks,</div>
<div class=""><br class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"> I currently do not understand that
behavior, but do not see any
problems<br class="">
with the code either. Are
there any possible
explanations for that
behavior?<br class="">
<br class="">
Thank you very much,<br
class="">
<br class="">
regards,<br class="">
<br class="">
Roland Richter<br class="">
<br class="">
</blockquote>
</div>
<br class="" clear="all">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr"
class="">
<div class="">What
most experimenters
take for granted
before they begin
their experiments
is infinitely more
interesting than
any results to
which their
experiments lead.<br
class="">
-- Norbert Wiener</div>
<div class=""><br
class="">
</div>
<div class=""><a
href="http://www.cse.buffalo.edu/~knepley/"
target="_blank"
moz-do-not-send="true" class="">https://www.cse.buffalo.edu/~knepley/</a><br
class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</div>
<br class="" clear="all">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most
experimenters take for granted
before they begin their
experiments is infinitely more
interesting than any results
to which their experiments
lead.<br class="">
-- Norbert Wiener</div>
<div class=""><br class="">
</div>
<div class=""><a
href="http://www.cse.buffalo.edu/~knepley/"
target="_blank"
moz-do-not-send="true"
class="">https://www.cse.buffalo.edu/~knepley/</a><br
class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br class="" clear="all">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="gmail_signature">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most experimenters take
for granted before they begin their
experiments is infinitely more
interesting than any results to which
their experiments lead.<br class="">
-- Norbert Wiener</div>
<div class=""><br class="">
</div>
<div class=""><a
href="http://www.cse.buffalo.edu/~knepley/"
target="_blank" moz-do-not-send="true"
class="">https://www.cse.buffalo.edu/~knepley/</a><br
class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<span id="cid:EFAE7E28-89EA-4363-863A-36124B074480"><main.cpp></span><span
id="cid:F1D708F0-E257-443A-8B4E-D003623603BF"><CMakeLists.txt></span></div>
</blockquote>
</div>
<br class="">
</blockquote>
</body>
</html>