<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>Dear Junchao,</div><div><br></div><div>Thank you for response and sorry for taking so long to answer back. </div><div>I cannot avoid using the nvidia tools. Gfortran is not mature for OpenACC and gives us problems when compiling our code.</div><div>What I have done to enable using the latest petsc is to create my own C code to call petsc. </div><div>I have little experience with c and it took me some time, but I can now use petsc 3.24.1 ;)</div><div><br></div><div>The behaviour remains the same as in my original email . </div><div>Parallel+GPU gives bad results. CPU(serial and parallel) and GPU serial all work ok and give the same result.</div><div><br></div><div>I have gone a bit into petsc comparing the CPU and GPU version with 2 mpi.</div><div>I see that the difference starts in </div><div>src/ksp/ksp/impls/cg/cg.c L170</div><div> PetscCall(KSP_PCApply(ksp, R, Z)); /* z <- Br */</div><div>I have printed the vectors R and Z and the norm dp.</div><div>R is identical on both CPU and GPU; but Z differs.</div><div>The correct value of dp (for the first time it enters) is 14.3014, while running on the GPU with 2 mpis it gives 14.7493.</div><div>If you wish I can send you prints I introduced in cg.c </div><div><br></div><div>The folder with the input files to run the case can be downloaded from <a href="https://urldefense.us/v3/__https://b2drop.eudat.eu/s/wKRQ4LK7RTKz2iQ__;!!G_uCfscf7eWS!Y_YIXIdrN81gDKgNed6V4icL3nN9OG62-ZsdnB1Bkc7iiGAoJ2riwbTzxJMnIROon3mXgiFVLnbH0RTlsXrLEAh7n_UO$">https://b2drop.eudat.eu/s/wKRQ4LK7RTKz2iQ</a></div><div><br></div><div>For submitting the gpu run I use </div><div>mpirun -np 2 --map-by ppr:4:node:PE=20 --report-bindings ./mn5_bind.sh /gpfs/scratch/bsc21/bsc021257/git/140-add-petsc/sod2d_gitlab/build_gpu/src/app_sod2d/sod2d ChannelFlowSolverIncomp.json</div><div><br></div><div>For the cpu run</div><div>mpirun -np 2 /gpfs/scratch/bsc21/bsc021257/git/140-add-petsc/sod2d_gitlab/build_cpu/src/app_sod2d/sod2d ChannelFlowSolverIncomp.json</div><div><br></div><div>Our code can be downloaded with :</div><div>git clone --recursive <a href="https://urldefense.us/v3/__https://gitlab.com/bsc_sod2d/sod2d_gitlab.git__;!!G_uCfscf7eWS!Y_YIXIdrN81gDKgNed6V4icL3nN9OG62-ZsdnB1Bkc7iiGAoJ2riwbTzxJMnIROon3mXgiFVLnbH0RTlsXrLEFjsBTIo$">https://gitlab.com/bsc_sod2d/sod2d_gitlab.git</a></div><div><br></div><div>-and the branch I am using with</div><div>git checkout 140-add-petsc</div><div><br></div><div>To use exactly the same commit I am using </div><div>git checkout 09a923c9b57e46b14ae54b935845d50272691ace</div><div><br></div><div><br></div><div>I am currently using: Currently Loaded Modules:</div><div> 1) nvidia-hpc-sdk/25.1 2) hdf5/1.14.1-2-nvidia-nvhpcx 3) cmake/3.25.1</div><div>I guess/hope similar modules should be available in any supercomputer.</div><div><br></div><div>To build the cpu version </div><div>mkdir build_cpu</div><div>cd build_cpu</div><div><br></div><div>export PETSC_INSTALL=/gpfs/scratch/bsc21/bsc021257/git/petsc_oct25/3241_cpu/hhinstal</div><div>export LD_LIBRARY_PATH=$PETSC_INSTALL/lib:$LD_LIBRARY_PATH</div><div>export LIBRARY_PATH=$PETSC_INSTALL/lib:$LIBRARY_PATH</div><div>export C_INCLUDE_PATH=$PETSC_INSTALL/include:$C_INCLUDE_PATH</div><div>export CPLUS_INCLUDE_PATH=$PETSC_INSTALL/include:$CPLUS_INCLUDE_PATH</div><div>export PKG_CONFIG_PATH=$PETSC_INSTALL/lib/pkgconfig:$PKG_CONFIG_PATH</div><div><br></div><div>cmake -DUSE_RP=8 -DUSE_PORDER=3 -DUSE_PETSC=ON -DUSE_GPU=OFF -DDEBUG_MODE=OFF ..</div><div>make -j 80</div><div><br></div><div>I have built petsc myself as follows</div><div><br></div><div>git clone -b release <a href="https://urldefense.us/v3/__https://gitlab.com/petsc/petsc.git__;!!G_uCfscf7eWS!Y_YIXIdrN81gDKgNed6V4icL3nN9OG62-ZsdnB1Bkc7iiGAoJ2riwbTzxJMnIROon3mXgiFVLnbH0RTlsXrLELP8U6d0$">https://gitlab.com/petsc/petsc.git</a> petsc</div><div>cd petsc</div><div>git checkout v3.24.1 </div><div>module purge</div><div>module load nvidia-hpc-sdk/25.1 hdf5/1.14.1-2-nvidia-nvhpcx cmake/3.25.1 </div><div>./configure --PETSC_DIR=/gpfs/scratch/bsc21/bsc021257/git/petsc_oct25/3241/petsc --prefix=/gpfs/scratch/bsc21/bsc021257/git/petsc_oct25/3241/hhinstal --with-fortran-bindings=0 --with-fc=0 --with-petsc-arch=linux-x86_64-opt --with-scalar-type=real --with-debugging=yes --with-64-bit-indices=1 --with-precision=single --download-hypre CFLAGS=-I/apps/ACC/HDF5/1.14.1-2/NVIDIA/NVHPCX/include CXXFLAGS= FCFLAGS= --with-shared-libraries=1 --with-mpi=1 --with-blacs-lib=/gpfs/apps/MN5/ACC/ONEAPI/2025.1/mkl/2025.1/lib/intel64/libmkl_blacs_openmpi_lp64.a --with-blacs-include=/gpfs/apps/MN5/ACC/ONEAPI/2025.1/mkl/2025.1/include --with-mpi-dir=/apps/ACC/NVIDIA-HPC-SDK/25.1/Linux_x86_64/25.1/comm_libs/12.6/hpcx/latest/ompi/ --download-ptscotch=yes --download-metis --download-parmetis</div><div>make all check</div><div>make install</div><div><br></div><div>-------------------</div><div>For the GPU version when configuring petsc I add : --with-cuda </div><div><br></div><div>I then change the export PETSC_INSTALL to </div><div>export PETSC_INSTALL=/gpfs/scratch/bsc21/bsc021257/git/petsc_oct25/3241/hhinstal</div><div>and repeat all other exports</div><div><br></div><div>mkdir build_gpu</div><div>cd build_gpu</div><div>cmake -DUSE_RP=8 -DUSE_PORDER=3 -DUSE_PETSC=ON -DUSE_GPU=ON -DDEBUG_MODE=OFF ..</div><div>make -j 80</div><div><br></div><div>As you can see from the submit instructions the executable is found in sod2d_gitlab/build_gpu/src/app_sod2d/sod2d</div><div><br></div><div>I hope I have not forgotten anything and my instructions are 'easy' to follow. If you have any issue do not doubt to contact me.</div><div>The wiki for our code can be found in <a href="https://urldefense.us/v3/__https://gitlab.com/bsc_sod2d/sod2d_gitlab/-/wikis/home__;!!G_uCfscf7eWS!Y_YIXIdrN81gDKgNed6V4icL3nN9OG62-ZsdnB1Bkc7iiGAoJ2riwbTzxJMnIROon3mXgiFVLnbH0RTlsXrLEA1vqPYk$">https://gitlab.com/bsc_sod2d/sod2d_gitlab/-/wikis/home</a></div><div><br></div><div>Best, </div><div><br></div><div>Herbert Owen</div><div> </div><div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>Herbert Owen<br>Senior Researcher, Dpt. Computer Applications in Science and Engineering<br>Barcelona Supercomputing Center (BSC-CNS)<br>Tel: +34 93 413 4038</div><div>Skype: herbert.owen<br><br><a href="https://urldefense.us/v3/__https://scholar.google.es/citations?user=qe5O2IYAAAAJ&hl=en__;!!G_uCfscf7eWS!Y_YIXIdrN81gDKgNed6V4icL3nN9OG62-ZsdnB1Bkc7iiGAoJ2riwbTzxJMnIROon3mXgiFVLnbH0RTlsXrLEAA5PwtO$">https://scholar.google.es/citations?user=qe5O2IYAAAAJ&hl=en</a></div><div><br></div></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<div><br><blockquote type="cite"><div>On 16 Oct 2025, at 18:30, Junchao Zhang <junchao.zhang@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><div dir="ltr"><div>Hi, Herbert,</div><div> I don't have much experience on OpenACC and PETSc CI doesn't have such tests. Could you avoid using nvfortran and instead use gfortran to compile your Fortran + OpenACC code? If you, then you can use the latest petsc code and make our debugging easier. </div><div> Also, could you provide us with a test and instructions to reproduce the problem?</div><div> </div><div> Thanks!</div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Thu, Oct 16, 2025 at 5:07 AM howen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Dear All,<div><br></div><div>I am interfacing our CFD code (Fortran + OpenACC) to Petsc. </div><div>Since we use OpenACC the natural choice for us is to use Nvidia´s nvhpc compiler. The Gnu compiler does not work well and we do not have access to the Cray compiler. </div><div><br></div><div>I already know that the latest version of Petsc does not compile with nvhpc, I am therefore using version 3.21. </div><div>I get good results on the CPU both in serial and parallel (MPI). However, the GPU implementation, that is what we are interested in, only work correctly for the serial version. In parallel, the results are different. Even for a CG solve. </div><div><br></div><div>I would like to know, if you have experience with the Nvidia compiler. I am particularly interested if you have already observed issues with it. Your opinion on whether to put further effort into trying to find a bug I may have introduced during the interfacing is highly appreciated.</div><div><br></div><div>Best,</div><div><br></div><div><div>
<div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div>Herbert Owen<br>Senior Researcher, Dpt. Computer Applications in Science and Engineering<br>Barcelona Supercomputing Center (BSC-CNS)<br>Tel: +34 93 413 4038</div><div>Skype: herbert.owen<br><br><a href="https://urldefense.us/v3/__https://scholar.google.es/citations?user=qe5O2IYAAAAJ&hl=en__;!!G_uCfscf7eWS!abuM7ozzUs7eISYBumHNxpvO2Tuy74KRM4-WWcunXHZVjQf1V032xQrCzTfC5vA_NM-35xMEZ9yJ8XK-3QFqjWBSWuUi$" target="_blank">https://scholar.google.es/citations?user=qe5O2IYAAAAJ&hl=en</a></div><div><br></div></div><br></div><br></div><br></div><br></div><br><br>
</div>
<br></div></div></blockquote></div>
</div></blockquote></div><br></body></html>