<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Hi </span><span style="display: inline !important; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Junchao,</span></div>
<div><span style="display: inline !important;"><br>
</span></div>
<div><span style="display: inline !important; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Thank you very much for your quick work. The simple build procedure now works.</span></div>
<div><span style="display: inline !important;"><br>
</span></div>
<div><span style="display: inline !important; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Jon</span></div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Junchao Zhang <junchao.zhang@gmail.com><br>
<b>Sent:</b> Sunday, December 19, 2021 6:38 PM<br>
<b>To:</b> petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Cc:</b> Jonathan D. Halverson <halverson@Princeton.EDU><br>
<b>Subject:</b> Re: [petsc-users] NVIDIA HPC SDK and complex data type</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Since it will take a while for NVIDIA to fix the bug in their NVCHPC 21.11 (December 2021), I added a workaround to the MR in petsc,
<a href="https://gitlab.com/petsc/petsc/-/merge_requests/4663">https://gitlab.com/petsc/petsc/-/merge_requests/4663</a></div>
<div>I tested it and it works with debugging (-O0) or no debugging (-O, or -O2). </div>
<br clear="all">
<div>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Sat, Dec 18, 2021 at 7:51 PM Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div style=""><br>
<div> Yes, Junchao deserves a bounty from NVIDIA for this find.</div>
<div><br>
<blockquote type="cite">
<div>On Dec 18, 2021, at 8:22 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div dir="ltr">On Sat, Dec 18, 2021 at 7:03 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br>
</div>
<div class="x_gmail_quote">
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div>I found it is a NVIDIA C/C++ compiler bug. I can reproduce it with</div>
</div>
</blockquote>
<div><br>
</div>
<div>Great find!</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div>
<div style="font-family:Menlo,Monaco,"Courier New",monospace; font-size:14px; line-height:21px; white-space:pre-wrap">
<div><span style="color:rgb(175,0,219)">#include</span><span style="color:rgb(0,0,255)">
</span><span style="color:rgb(163,21,21)"><stdlib.h></span></div>
<div><span style="color:rgb(175,0,219)">#include</span><span style="color:rgb(0,0,255)">
</span><span style="color:rgb(163,21,21)"><stdio.h></span><span style="color:rgb(163,21,21)"><br>
</span></div>
<div><span style="color:rgb(175,0,219)">#include</span><span style="color:rgb(0,0,255)">
</span><span style="color:rgb(163,21,21)"><complex.h></span></div>
<br>
<div><span style="color:rgb(0,0,255)">typedef</span> <span style="color:rgb(0,0,255)">
double</span> <span style="color:rgb(0,0,255)">_Complex</span> PetscScalar;</div>
<div><span style="color:rgb(0,0,255)">typedef</span> <span style="color:rgb(0,0,255)">
struct</span> {</div>
<div><span style="color:rgb(0,0,255)">int</span> row;</div>
<div>PetscScalar *valaddr;</div>
<div>} MatEntry2;</div>
<br>
<div><span style="color:rgb(0,0,255)">int</span> <span style="color:rgb(121,94,38)">
main</span>(<span style="color:rgb(0,0,255)">int</span> <span style="color:rgb(0,16,128)">
arc</span>, <span style="color:rgb(0,0,255)">char</span>** <span style="color:rgb(0,16,128)">
argv</span>)</div>
<div>{</div>
<div><span style="color:rgb(0,0,255)">int</span> i=<span style="color:rgb(9,134,88)">2</span>;</div>
<div>MatEntry2 *Jentry2 = (MatEntry2*)<span style="color:rgb(121,94,38)">malloc</span>(<span style="color:rgb(9,134,88)">64</span>*<span style="color:rgb(0,0,255)">sizeof</span>(MatEntry2));</div>
<div>PetscScalar a=<span style="color:rgb(9,134,88)">1</span>, b=<span style="color:rgb(9,134,88)">1</span>;</div>
<div><br>
</div>
<div>
<div style="line-height:21px; white-space:pre-wrap">
<div><span style="color:rgb(121,94,38)">printf</span>(<span style="color:rgb(163,21,21)">"sizeof(MatEntry2)=</span><span style="color:rgb(0,16,128)">%lu</span><span style="color:rgb(238,0,0)">\n</span><span style="color:rgb(163,21,21)">"</span>,<span style="color:rgb(0,0,255)">sizeof</span>(MatEntry2));</div>
</div>
</div>
<div><span style="color:rgb(0,16,128)">Jentry2</span>[<span style="color:rgb(9,134,88)">2</span>].<span style="color:rgb(0,16,128)">valaddr</span> = (PetscScalar*)<span style="color:rgb(121,94,38)">malloc</span>(<span style="color:rgb(9,134,88)">16</span>*<span style="color:rgb(0,0,255)">sizeof</span>(PetscScalar));</div>
<div>*(<span style="color:rgb(0,16,128)">Jentry2</span>[i].<span style="color:rgb(0,16,128)">valaddr</span>) = a*b; // Segfault</div>
<div><br>
</div>
<div><span style="color:rgb(121,94,38)">free</span>(<span style="color:rgb(0,16,128)">Jentry2</span>[<span style="color:rgb(9,134,88)">2</span>].<span style="color:rgb(0,16,128)">valaddr</span>);</div>
<div><span style="color:rgb(121,94,38)">free</span>(Jentry2);</div>
<div><span style="color:rgb(175,0,219)">return</span> <span style="color:rgb(9,134,88)">
0</span>;</div>
<div>}</div>
</div>
</div>
<div><br>
</div>
<font face="monospace">$ nvc -O0 -o test test.c<br>
$ ./test<br>
sizeof(MatEntry2)=16<br>
Segmentation fault (core dumped)</font>
<div><br>
</div>
If I change <font face="monospace">*(Jentry2[i].valaddr) = a*b;</font> to<br>
<br>
<font face="monospace">PetscScalar *p = Jentry2[2].valaddr;<br>
*p = a*b;</font>
<div><br>
</div>
<div>Then the code works fine. Using -O0 to -O2 will also avoid this error for this simple test, but not for PETSc. In PETSc, I could apply the above silly trick, but I am not sure it is worth it. We should instead report it to NVIDIA.</div>
<div><br>
</div>
<div>Looking at the assembly code for the segfault line, we can find the problem</div>
<font face="monospace"> movslq 52(%rsp), %rcx<br>
movq 40(%rsp), %rax<br>
movq 8(%rax,%rcx,8), %rax // Here %rax = &Jentry2, %rcx = i; The instruction wrongly calculates <span style="color:rgb(0,16,128)">Jentry2</span>[<span style="color:rgb(9,134,88)">2</span>].<span style="color:rgb(0,16,128)">valaddr as (%rax + %rcx*8)+8,
which should instead be </span><span style="color:rgb(0,16,128)">(%rax + %rcx*16)+8</span><br>
vmovsd %xmm1, 8(%rax)<br>
</font>
<div><font face="monospace"> vmovsd %xmm0, (%rax) </font></div>
<div><br>
</div>
<div>--Junchao Zhang<br>
</div>
<br>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Fri, Dec 17, 2021 at 7:58 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">Hi, Jon,
<div> I could reproduce the error exactly. I will have a look.</div>
<div> Thanks for reporting.<br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Fri, Dec 17, 2021 at 2:56 PM Jonathan D. Halverson <<a href="mailto:halverson@princeton.edu" target="_blank">halverson@princeton.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Hello,</span><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">We are unable to build PETSc using the <span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">NVIDIA HPC SDK and
<span style="background-color:rgb(255,255,255); display:inline">--with-scalar-type=complex</span>. Below is our procedure:</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<div style="color:rgb(32,31,30); font-size:15px; margin:0px"><span style="margin:0px; font-family:Calibri,Helvetica,sans-serif; font-size:12pt; font-variant-ligatures:no-common-ligatures">$ module load<span> </span></span><span style="margin:0px; font-family:Calibri,Helvetica,sans-serif; font-size:12pt; font-variant-ligatures:no-common-ligatures"><span style="margin:0px">nvhpc</span>/21.11</span></div>
<div style="color:rgb(32,31,30); font-size:15px; margin:0px"><span style="margin:0px; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">$ module load openmpi/<span style="margin:0px">nvhpc</span>-21.11/4.1.2/64</span></div>
<div style="margin:0px; font-size:15px; color:rgb(32,31,30); min-height:21px"><span style="margin:0px; font-family:Calibri,Helvetica,sans-serif; font-size:12pt; font-variant-ligatures:no-common-ligatures">$ git clone -b release
<a href="https://gitlab.com/petsc/petsc.git" target="_blank">https://gitlab.com/petsc/petsc.git</a> petsc; </span><span style="margin:0px; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">cd petsc</span><br>
<span style="margin:0px; font-variant-ligatures:no-common-ligatures"></span></div>
<div style="color:rgb(32,31,30); font-size:15px; margin:0px"><span style="margin:0px; font-family:Calibri,Helvetica,sans-serif; font-size:12pt; font-variant-ligatures:no-common-ligatures">$ ./configure --with-debugging=1 --with-scalar-type=complex PETSC_ARCH=openmpi-power</span></div>
<div style="color:rgb(32,31,30); font-size:15px; margin:0px"><span style="margin:0px; font-family:Calibri,Helvetica,sans-serif; font-size:12pt; font-variant-ligatures:no-common-ligatures">$ make PETSC_DIR=/home/$USER/software/petsc PETSC_ARCH=openmpi-power
all</span></div>
<div style="color:rgb(32,31,30); font-size:15px; margin:0px"><span style="margin:0px; font-family:Calibri,Helvetica,sans-serif; font-size:12pt; font-variant-ligatures:no-common-ligatures">$ make PETSC_DIR=/home/$USER/software/petsc PETSC_ARCH=openmpi-power
check</span></div>
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">"make check" fails with a segmentation fault when running ex19. The fortran test ex5f passes.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><span style="background-color:rgb(255,255,255); display:inline">The procedure above fails on x86_64 and POWER both running RHEL8. It also fails using nvhpc 20.7.</span><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">The procedure above works for "real" instead of "complex".</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">A "hello world" MPI code using a complex data type works with our nvhpc modules.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">The procedure above works successfully when GCC and an Open MPI library built using GCC is used.</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">The only trouble is the combination of PETSc with nvhpc and complex. Any known issues?</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">The build log for the procedure above is here:</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><a href="https://tigress-web.princeton.edu/~jdh4/petsc_nvhpc_complex_17dec2021.log" target="_blank">https://tigress-web.princeton.edu/~jdh4/petsc_nvhpc_complex_17dec2021.log</a><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Jon</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</body>
</html>