<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div> Yes, Junchao deserves a bounty from NVIDIA for this find.</div><div><br class=""><blockquote type="cite" class=""><div class="">On Dec 18, 2021, at 8:22 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Sat, Dec 18, 2021 at 7:03 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="">I found it is a NVIDIA C/C++ compiler bug. I can reproduce it with</div></div></blockquote><div class=""><br class=""></div><div class="">Great find!</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class=""><div style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 14px; line-height: 21px; white-space: pre-wrap;" class=""><div class=""><span style="color:rgb(175,0,219)" class="">#include</span><span style="color:rgb(0,0,255)" class=""> </span><span style="color:rgb(163,21,21)" class=""><stdlib.h></span></div><div class=""><span style="color:rgb(175,0,219)" class="">#include</span><span style="color:rgb(0,0,255)" class=""> </span><span style="color:rgb(163,21,21)" class=""><stdio.h></span><span style="color:rgb(163,21,21)" class=""><br class=""></span></div><div class=""><span style="color:rgb(175,0,219)" class="">#include</span><span style="color:rgb(0,0,255)" class=""> </span><span style="color:rgb(163,21,21)" class=""><complex.h></span></div><br class=""><div class=""><span style="color:rgb(0,0,255)" class="">typedef</span> <span style="color:rgb(0,0,255)" class="">double</span> <span style="color:rgb(0,0,255)" class="">_Complex</span> PetscScalar;</div><div class=""><span style="color:rgb(0,0,255)" class="">typedef</span> <span style="color:rgb(0,0,255)" class="">struct</span> {</div><div class=""> <span style="color:rgb(0,0,255)" class="">int</span> row;</div><div class=""> PetscScalar *valaddr;</div><div class="">} MatEntry2;</div><br class=""><div class=""><span style="color:rgb(0,0,255)" class="">int</span> <span style="color:rgb(121,94,38)" class="">main</span>(<span style="color:rgb(0,0,255)" class="">int</span> <span style="color:rgb(0,16,128)" class="">arc</span>, <span style="color:rgb(0,0,255)" class="">char</span>** <span style="color:rgb(0,16,128)" class="">argv</span>)</div><div class="">{</div><div class=""> <span style="color:rgb(0,0,255)" class="">int</span> i=<span style="color:rgb(9,134,88)" class="">2</span>;</div><div class=""> MatEntry2 *Jentry2 = (MatEntry2*)<span style="color:rgb(121,94,38)" class="">malloc</span>(<span style="color:rgb(9,134,88)" class="">64</span>*<span style="color:rgb(0,0,255)" class="">sizeof</span>(MatEntry2));</div><div class=""> PetscScalar a=<span style="color:rgb(9,134,88)" class="">1</span>, b=<span style="color:rgb(9,134,88)" class="">1</span>;</div><div class=""><br class=""></div><div class=""><div style="line-height:21px;white-space:pre-wrap" class=""><div class=""> <span style="color:rgb(121,94,38)" class="">printf</span>(<span style="color:rgb(163,21,21)" class="">"sizeof(MatEntry2)=</span><span style="color:rgb(0,16,128)" class="">%lu</span><span style="color:rgb(238,0,0)" class="">\n</span><span style="color:rgb(163,21,21)" class="">"</span>,<span style="color:rgb(0,0,255)" class="">sizeof</span>(MatEntry2));</div></div></div><div class=""> <span style="color:rgb(0,16,128)" class="">Jentry2</span>[<span style="color:rgb(9,134,88)" class="">2</span>].<span style="color:rgb(0,16,128)" class="">valaddr</span> = (PetscScalar*)<span style="color:rgb(121,94,38)" class="">malloc</span>(<span style="color:rgb(9,134,88)" class="">16</span>*<span style="color:rgb(0,0,255)" class="">sizeof</span>(PetscScalar));</div><div class=""> *(<span style="color:rgb(0,16,128)" class="">Jentry2</span>[i].<span style="color:rgb(0,16,128)" class="">valaddr</span>) = a*b; // Segfault</div><div class=""><br class=""></div><div class=""> <span style="color:rgb(121,94,38)" class="">free</span>(<span style="color:rgb(0,16,128)" class="">Jentry2</span>[<span style="color:rgb(9,134,88)" class="">2</span>].<span style="color:rgb(0,16,128)" class="">valaddr</span>);</div><div class=""> <span style="color:rgb(121,94,38)" class="">free</span>(Jentry2);</div><div class=""> <span style="color:rgb(175,0,219)" class="">return</span> <span style="color:rgb(9,134,88)" class="">0</span>;</div><div class="">}</div></div></div><div class=""><br class=""></div><font face="monospace" class="">$ nvc -O0 -o test test.c<br class="">$ ./test<br class="">sizeof(MatEntry2)=16<br class="">Segmentation fault (core dumped)</font><div class=""><br class=""></div>If I change <font face="monospace" class="">*(Jentry2[i].valaddr) = a*b;</font> to<br class=""><br class=""><font face="monospace" class="">PetscScalar *p = Jentry2[2].valaddr;<br class="">*p = a*b;</font><div class=""><br class=""></div><div class="">Then the code works fine. Using -O0 to -O2 will also avoid this error for this simple test, but not for PETSc. In PETSc, I could apply the above silly trick, but I am not sure it is worth it. We should instead report it to NVIDIA.</div><div class=""><br class=""></div><div class="">Looking at the assembly code for the segfault line, we can find the problem</div><font face="monospace" class=""> movslq 52(%rsp), %rcx<br class=""> movq 40(%rsp), %rax<br class=""> movq 8(%rax,%rcx,8), %rax // Here %rax = &Jentry2, %rcx = i; The instruction wrongly calculates <span style="color:rgb(0,16,128)" class="">Jentry2</span>[<span style="color:rgb(9,134,88)" class="">2</span>].<span style="color:rgb(0,16,128)" class="">valaddr as (%rax + %rcx*8)+8, which should instead be </span><span style="color:rgb(0,16,128)" class="">(%rax + %rcx*16)+8</span><br class=""> vmovsd %xmm1, 8(%rax)<br class=""></font><div class=""><font face="monospace" class=""> vmovsd %xmm0, (%rax) </font></div><div class=""><br class=""></div><div class="">--Junchao Zhang<br class=""></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 17, 2021 at 7:58 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Hi, Jon,<div class=""> I could reproduce the error exactly. I will have a look.</div><div class=""> Thanks for reporting.<br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 17, 2021 at 2:56 PM Jonathan D. Halverson <<a href="mailto:halverson@princeton.edu" target="_blank" class="">halverson@princeton.edu</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
<span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">Hello,</span><br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
We are unable to build PETSc using the <span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">NVIDIA HPC SDK and
<span style="background-color:rgb(255,255,255);display:inline" class="">
--with-scalar-type=complex</span>. Below is our procedure:</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class=""><div style="color: rgb(32, 31, 30); font-size: 15px; margin: 0px;" class="">
<span style="margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; font-variant-ligatures: no-common-ligatures;" class="">$ module load<span class=""> </span></span><span style="margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; font-variant-ligatures: no-common-ligatures;" class=""><span style="margin:0px" class="">nvhpc</span>/21.11</span></div><div style="color: rgb(32, 31, 30); font-size: 15px; margin: 0px;" class="">
<span style="margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">$ module load openmpi/<span style="margin:0px" class="">nvhpc</span>-21.11/4.1.2/64</span></div>
<div style="margin:0px;font-size:15px;color:rgb(32,31,30);min-height:21px" class="">
<span style="margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; font-variant-ligatures: no-common-ligatures;" class="">$ git clone -b release <a href="https://gitlab.com/petsc/petsc.git" target="_blank" class="">https://gitlab.com/petsc/petsc.git</a> petsc; </span><span style="margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">cd
petsc</span><br class="">
<span style="margin:0px;font-variant-ligatures:no-common-ligatures" class=""></span></div><div style="color: rgb(32, 31, 30); font-size: 15px; margin: 0px;" class="">
<span style="margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; font-variant-ligatures: no-common-ligatures;" class="">$ ./configure --with-debugging=1 --with-scalar-type=complex PETSC_ARCH=openmpi-power</span></div><div style="color: rgb(32, 31, 30); font-size: 15px; margin: 0px;" class="">
<span style="margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; font-variant-ligatures: no-common-ligatures;" class="">$ make PETSC_DIR=/home/$USER/software/petsc PETSC_ARCH=openmpi-power all</span></div><div style="color: rgb(32, 31, 30); font-size: 15px; margin: 0px;" class="">
<span style="margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; font-variant-ligatures: no-common-ligatures;" class="">$ make PETSC_DIR=/home/$USER/software/petsc PETSC_ARCH=openmpi-power check</span></div>
<br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
"make check" fails with a segmentation fault when running ex19. The fortran test ex5f passes.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
<span style="background-color:rgb(255,255,255);display:inline" class="">The procedure above fails on x86_64 and POWER both running RHEL8. It also fails using nvhpc 20.7.</span><br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
The procedure above works for "real" instead of "complex".</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
A "hello world" MPI code using a complex data type works with our nvhpc modules.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
The procedure above works successfully when GCC and an Open MPI library built using GCC is used.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
The only trouble is the combination of PETSc with nvhpc and complex. Any known issues?</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
The build log for the procedure above is here:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
<a href="https://tigress-web.princeton.edu/~jdh4/petsc_nvhpc_complex_17dec2021.log" target="_blank" class="">https://tigress-web.princeton.edu/~jdh4/petsc_nvhpc_complex_17dec2021.log</a><br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
<br class="">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">
Jon</div>
</div>
</blockquote></div>
</blockquote></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></body></html>