<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I wrote the following dummy MPI program, and followed the wiki page to generate the suppression file. It helps a bit, but still generates a lot of unwanted ones. For a 8-process run, I got the output file with over 45000 lines. </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Is this mainly due to MPI program? I remember using Linux version with sequential program is very clean. </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Jeff -- if you succeed with building MPICH out Cori, I'd like to use that to simply the use of valgrind.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Thanks,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Sherry</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo"><b>#include</b> <stdio.h></p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo"><b>#include</b> <mpi.h></p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo;min-height:19px"><br></p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo"><b>// compile: cc test.c $VALGRIND_MPI_LINK </b></p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo"><b>// run: srun -n 1 valgrind --leak-check=full --gen-suppressions=all ./a.out </b></p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo;min-height:19px"><br></p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo"><span style="color:rgb(255,255,255);background-color:rgb(0,0,0)"><b>main</b></span>(<span style="text-decoration:underline"><b>int</b></span> <b>argc</b>, <span style="text-decoration:underline"><b>char</b></span> *<b>argv</b>[])</p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo">{</p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo"> <span style="text-decoration:underline"><b>int</b></span> <b>iam</b>, <b>nprocs</b>;</p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo"> MPI_Init( &argc, &argv );</p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo;min-height:19px"><br></p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo"> MPI_Comm_size( MPI_COMM_WORLD, &nprocs );</p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo"> MPI_Comm_rank( MPI_COMM_WORLD, &iam );</p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo"> printf("iam %d, nprocs %d\n", iam, nprocs); fflush(stdout);</p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo;min-height:19px"><br></p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo"> MPI_Finalize();</p>
<p style="margin:0px;font-size:16px;line-height:normal;font-family:Menlo">}</p><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 16, 2016 at 5:42 PM, Jeff Hammond <span dir="ltr"><<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Sherry, Satish:<div><br></div><div>Building MPI from source on Cray machines is not trivial, because Cray MPI is proprietary.</div><div><br></div><div>However building MPICH on Cray is actually possible now as of a month or two ago thanks to libfabric, but there are approximately two people on earth who have done it successfully. </div><div><br></div><div>If it is critical for debugging to have this at NERSC, I'll share the Cori MPI build details once I've got everything in order. <span></span></div><div><br></div><div>Jeff<div><div class="h5"><br><br>On Saturday, January 16, 2016, Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Sat, 16 Jan 2016, Xiaoye S. Li wrote:<br>
<br>
> By the way, I have a question about valgrind. Lately I have been chasing<br>
> a possible memory corruption on Cori at NERSC. Valgrind gave LOTS OF<br>
> internal system-related warnings, such as:<br>
><br>
> ==39059== Conditional jump or move depends on uninitialised value(s)<br>
> ==39059== at 0xD5E336: __register_atfork (register-atfork.c:119)<br>
> ==39059== by 0xD5E418: __libc_pthread_init (libc_pthread_init.c:48)<br>
> ==39059== by 0x6629E1: __pthread_initialize_minimal (nptl-init.c:462)<br>
> ==39059== by 0xD14BDA: (below main) (libc-start.c:152)<br>
> ==39059==<br>
> ==39057== by 0xD5E418: __libc_pthread_init (libc_pthread_init.c:48)<br>
> ==39057== by 0x6629E1: __pthread_initialize_minimal (nptl-init.c:462)<br>
> ==39057== by 0xD14BDA: (below main) (libc-start.c:152)<br>
><br>
> This has nothing do do with my program. Do you know a way to NOT to print<br>
> those? I got a huge file filled with these, hard to find out the real<br>
> error.<br>
<br>
You can ask valgrind to create a supression file.<br>
<br>
[then edit this file - and format it to include only the stuff that you want to supress]<br>
<br>
And then - use this file for your next run - to catch the actual issues.<br>
<br>
<a href="https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto" target="_blank">https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto</a> has some of this info..<br>
<br>
<br>
We normally recommend using valgrind on linux with mpich built with<br>
'--enable-g=meminit' [--download-mpich option with petsc configure<br>
defaults to this mode] - so that its valgrind clean - and not bother<br>
with supression files.<br>
<br>
Satish</blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><br><br>-- <br>Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a><br>
</font></span></blockquote></div><br></div>