<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><blockquote type="cite" class=""><span class="" style="display: inline !important;">MPICH need to be built with the option --enable-g=meminit for it to be valgrind clean.</span></blockquote>I see. Two questions:<div class=""><br class=""><div class="">1. I am using Jeds docker image for MPICH (actually this is the main reason for using Jeds image, as configuring and building MPICH takes an absolute age on docker), will this re-build the MPICH already installed in the image or download and build an entirely new MPICH?</div><div class=""><br class=""><div class="">2. (More a question for Jed) If I am using Jeds docker image anyways and we have CI/CD builds also using his docker images (correct me if I am wrong here) and a valgrind build in stage 3 why not include those build arguments in the MPICH included in the images? I think the valgrind build for CI/CD actually downloads and builds MPICH through petsc configure, is this because of the errors I detailed below?</div><div class=""><br class=""></div><div class=""><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div>Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: (312) 694-3391</div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Jun 2, 2020, at 11:59 AM, Satish Balay <<a href="mailto:balay@mcs.anl.gov" class="">balay@mcs.anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">MPICH need to be built with the option --enable-g=meminit for it to be valgrind clean.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">--download-mpich does this [among other things that are useful during software developement]. Pre-configured MPICH is not likely to do this.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">You can verify with: mpichversion</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">You can prebuild MPICH using PETSc with:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">./configure --prefix=$HOME/soft/mpich --download-mpich CFLAGS= FFLAGS= CXXFLAGS= COPTFLAGS= CXXOPTFLAGS= FOPTFLAGS=</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">And make this your default pre-installed MPI. [by adding $HOME/soft/mpich to PATH]</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Satish</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">On Tue, 2 Jun 2020, Jacob Faibussowitsch wrote:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Yes I am using the pre-loaded MPICH from the docker image. Further proof from configure<br class=""><br class="">#define PETSC_HAVE_MPICH_NUMVERSION 30302300<br class="">#define PETSC_HAVE_MPIEXEC_ENVIRONMENTAL_VARIABLE MPIR_CVAR_CH3<br class=""><br class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: (312) 694-3391<br class=""><br class=""><blockquote type="cite" class="">On Jun 2, 2020, at 11:35 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""><br class="">I guess Jacob already used MPICH, since MPIDI_CH3_EagerContigShortSend() is from MPICH.<span class="Apple-converted-space"> </span><br class=""><br class="">--Junchao Zhang<br class=""><br class=""><br class="">On Tue, Jun 2, 2020 at 9:38 AM Satish Balay via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-dev@mcs.anl.gov" class="">mailto:petsc-dev@mcs.anl.gov</a>>> wrote:<br class="">use --download-mpich for valgrind.<br class=""><br class=""><a href="https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><<a href="https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a>><br class=""><br class="">Satish<br class=""><br class="">On Tue, 2 Jun 2020, Karl Rupp wrote:<br class=""><br class=""><blockquote type="cite" class="">Hi Jacob,<br class=""><br class="">the recommendation in the past was to use MPICH as it is (was?)<br class="">valgrind-clean. Which MPI do you use? OpenMPI used to have these kinds of<br class="">issues. (My information might be outdated)<br class=""><br class="">Best regards,<br class="">Karli<br class=""><br class="">On 6/2/20 2:43 AM, Jacob Faibussowitsch wrote:<br class=""><blockquote type="cite" class="">Hello All,<br class=""><br class="">TL;DR: valgrind always complains about "Syscall param write(buf) points to<br class="">uninitialised byte(s)” for a LOT of MPI operations in petsc code, making<br class="">debugging using valgrind fairly annoying since I have to sort through a ton<br class="">of unrelated stuff. I have built valgrind from source, used apt install<br class="">valgrind, apt install valgrind-mpi to no avail.<br class=""><br class="">I am using valgrind from docker. Dockerfile is attached below as well. I<br class="">have been unsuccessfully trying to resolve these local valgrind errors, but<br class="">I am running out of ideas. Googling the issue has also not provided entirely<br class="">applicable solutions. Here is an example of the error:<br class=""><br class="">$ make -f gmakefile test VALGRIND=1<br class="">...<br class="">#==54610== Syscall param write(buf) points to uninitialised byte(s)<br class="">#==54610== at 0x6F63317: write (write.c:26)<br class="">#==54610== by 0x9056AC9: MPIDI_CH3I_Sock_write (in<span class="Apple-converted-space"> </span><br class="">/usr/local/lib/libmpi.so.12.1.8)<br class="">#==54610== by 0x9059FCD: MPIDI_CH3_iStartMsg (in<br class="">/usr/local/lib/libmpi.so.12.1.8)<br class="">#==54610== by 0x903F298: MPIDI_CH3_EagerContigShortSend (in<br class="">/usr/local/lib/libmpi.so.12.1.8)<br class="">#==54610== by 0x9049479: MPID_Send (in /usr/local/lib/libmpi.so.12.1.8)<br class="">#==54610== by 0x8FC9B2A: MPIC_Send (in /usr/local/lib/libmpi.so.12.1.8)<br class="">#==54610== by 0x8F86F2E: MPIR_Bcast_intra_binomial (in<span class="Apple-converted-space"> </span><br class="">/usr/local/lib/libmpi.so.12.1.8)<br class="">#==54610== by 0x8EE204E: MPIR_Bcast_intra_auto (in<br class="">/usr/local/lib/libmpi.so.12.1.8)<br class="">#==54610== by 0x8EE21F4: MPIR_Bcast_impl (in<br class="">/usr/local/lib/libmpi.so.12.1.8)<br class="">#==54610== by 0x8F887FB: MPIR_Bcast_intra_smp (in<br class="">/usr/local/lib/libmpi.so.12.1.8)<br class="">#==54610== by 0x8EE206E: MPIR_Bcast_intra_auto (in<br class="">/usr/local/lib/libmpi.so.12.1.8)<br class="">#==54610== by 0x8EE21F4: MPIR_Bcast_impl (in<br class="">/usr/local/lib/libmpi.so.12.1.8)<br class="">#==54610== by 0x8EE2A6F: PMPI_Bcast (in /usr/local/lib/libmpi.so.12.1.8)<br class="">#==54610== by 0x4B377B8: PetscOptionsInsertFile (options.c:525)<br class="">#==54610== by 0x4B39291: PetscOptionsInsert (options.c:672)<br class="">#==54610== by 0x4B5B1EF: PetscInitialize (pinit.c:996)<br class="">#==54610== by 0x10A6BA: main (ex9.c:75)<br class="">#==54610== Address 0x1ffeffa944 is on thread 1's stack<br class="">#==54610== in frame #3, created by MPIDI_CH3_EagerContigShortSend (???:)<br class="">#==54610== Uninitialised value was created by a stack allocation<br class="">#==54610== at 0x903F200: MPIDI_CH3_EagerContigShortSend (in<span class="Apple-converted-space"> </span><br class="">/usr/local/lib/libmpi.so.12.1.8)<br class=""><br class="">There are probably 20 such errors every single time, regardless of what code<br class="">is being run. I have tried using apt install valgrind, apt install<br class="">valgrind-mpi, and building valgrind from source:<br class=""><br class=""># VALGRIND<br class="">WORKDIR /<br class="">RUN git clone<span class="Apple-converted-space"> </span><a href="git://sourceware.org/git/valgrind.git" class="">git://sourceware.org/git/valgrind.git</a><span class="Apple-converted-space"> </span><<a href="http://sourceware.org/git/valgrind.git" class="">http://sourceware.org/git/valgrind.git</a>><br class="">WORKDIR /valgrind<br class="">RUN git pull<br class="">RUN ./autogen.sh<br class="">RUN ./configure --with-mpicc=/usr/local/bin/mpicc<br class="">RUN make -j 5<br class="">RUN make install<br class=""><br class="">None of the those approaches lead to these errors disappearing. Perhaps I am<br class="">missing some funky MPI args?<br class=""><br class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: (312) 694-3391</blockquote></blockquote></blockquote></blockquote></div></blockquote></div><br class=""></div></div></div></body></html>