<div dir="ltr">Hi Phil,<div><br></div><div>Thanks for the suggestion. I solved this issue finally </div><div>by recompiling zlib.</div><div><br></div><div>Teng</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 21, 2017 at 1:05 PM, Phil Carns <span dir="ltr"><<a href="mailto:carns@mcs.anl.gov" target="_blank">carns@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <div class="m_-8621407278620756273moz-cite-prefix">It might be helpful to compare the
      mpich configure/compile lines between your two configurations too,
      in case there is some difference in mpich libraries that is
      triggering an issue.<br>
      <br>
      thanks,<br>
      -Phil<div><div class="h5"><br>
      <br>
      On 07/20/2017 05:35 PM, Shane Snyder wrote:<br>
    </div></div></div><div><div class="h5">
    <blockquote type="cite">
      
      <br>
      <div class="m_-8621407278620756273moz-cite-prefix">On 07/20/2017 04:19 PM, Shane Snyder
        wrote:<br>
      </div>
      <blockquote type="cite"> Hi
        Teng,<br>
        <br>
        First of all, I actually just got darshan 3.1.4 to properly
        generate a log when using static linking. I'm not sure what was
        leading to the problem I had earlier, but after reconfiguring
        and rebuilding the library, it works for me now. I also
        confirmed the nightly tests were working correctly -- the log
        files were going to another spot that I wasn't expecting, so
        missed them the first time around. You may want to try something
        similar -- just rebuilding everything from scratch and trying
        again. I've also run into this problem in the past whenever I
        was accidentally pointing to an mpich build that defaults to
        using shared libraries rather than the static ones, so that
        could be another thing to check. <br>
        <br>
        As for the error you are getting with Darshan 3.1.3, I've
        actually never seen someone hit that before. That error would be
        due to the libz compression functions (deflate) returning some
        sort of error. Not exactly sure what could cause that, but is
        the application opening a really large amount of files or
        something like that? I suppose it's possible the compression
        routines could be running out of memory if there is too much log
        data they are trying to compress, but have never seen that
        happen before. If you aren't already, it might be worth trying a
        test program that does very basic I/O just to confirm that we
        can get Darshan working. One that we frequently use is
        mpi-io-test -- there is a version of this code in the darshan
        source @ darshan-test/regression/test-<wbr>cases/src. You could try
        just building that and running it (it doesn't take any command
        line parameters) to see if that works?<br>
      </blockquote>
      <br>
      Oh, actually I just noticed in your emails that you were using
      mpi-io-test? That is pretty bizarre...<br>
      <br>
      What system are you running this on? A cluster? Your own personal
      machine? If possible, you might try running somewhere else so we
      can confirm the issue is some problem with the environment on the
      system you have been using... I can't think of why the deflate
      routines would fail when running something simple like
      mpi-io-test. But if needed, we could probably get you a patch or
      something that tries to give us more details on the libz error you
      are seeing. <br>
      <br>
      --Shane<br>
      <br>
      <blockquote type="cite"> <br>
        Thanks,<br>
        --Shane<br>
        <br>
        <div class="m_-8621407278620756273moz-cite-prefix">On 07/20/2017 03:52 PM, Teng Wang
          wrote:<br>
        </div>
        <blockquote type="cite">
          <div dir="ltr">Hi Shane,
            <div><br>
            </div>
            <div>Thanks for your help on this. I tried v3.1.3 and
              compiled successfully. </div>
            <div>But when I ran the application, there was no *.log
              generated under the </div>
            <div>log directory, and I also got a warning from the
              application:</div>
            <div>
              <p style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(175,173,36)"> </span><span style="font-variant-ligatures:no-common-ligatures">darshan
                  library warning: unable to compress job data</span></p>
              <p style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><br>
              </p>
            </div>
            <div>May I know how to fix this?</div>
            <div><br>
            </div>
            <div>Thanks,</div>
            <div>Teng</div>
          </div>
          <div class="gmail_extra"><br>
            <div class="gmail_quote">On Thu, Jul 20, 2017 at 11:51 AM,
              Shane Snyder <span dir="ltr"><<a href="mailto:ssnyder@mcs.anl.gov" target="_blank">ssnyder@mcs.anl.gov</a>></span>
              wrote:<br>
              <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                <div text="#000000" bgcolor="#FFFFFF"> Hi Teng,<br>
                  <br>
                  Thanks for reporting this. This is a really strange
                  issue, but I can actually reproduce it, too. I'm going
                  to keep digging and see if I can find the problem.
                  Will keep you updated.<br>
                  <br>
                  The strange thing is that I was able to use version
                  3.1.4 successfully right before making the release
                  available (i.e., the log @
                  darshan-test/example-output/mp<wbr>i-io-test-x86_64-3.1.4.darshan
                  was generated using version 3.1.4 with static
                  linking). Also strange is that our nightly testing is
                  not catching this issue...<br>
                  <br>
                  FYI, version 3.1.3 and earlier work fine for me if you
                  really need a quick resolution -- version 3.1.4 just
                  fixed a couple of non-critical bugs in some darshan
                  log parsing utilities, so there shouldn't be much
                  functionally different between the versions.<br>
                  <br>
                  --Shane
                  <div>
                    <div class="m_-8621407278620756273h5"><br>
                      <br>
                      <div class="m_-8621407278620756273m_6259078016213707887moz-cite-prefix">On
                        07/20/2017 12:43 PM, Teng Wang wrote:<br>
                      </div>
                    </div>
                  </div>
                  <blockquote type="cite">
                    <div>
                      <div class="m_-8621407278620756273h5">
                        <div dir="ltr">
                          <p>Hi,</p>
                          <p><br>
                          </p>
                          <p>I'm having trouble when trying to
                            statically link </p>
                          <p>Darshan with mpich3.3 applications
                            following the instructions </p>
                          <p>here. Could you give me any suggestion on
                            how to fix it? </p>
                          <p><br>
                          </p>
                          <p><span style="text-decoration:underline"><a href="http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-runtime.html" target="_blank">http://www.mcs.anl.gov/researc<wbr>h/projects/darshan/docs/<wbr>darshan-runtime.html</a></span></p>
                          <p><br>
                          </p>
                          <p>Here is my steps:</p>
                          <p>1. Compile and install darshan 3.1.4</p>
                          <p>tar -xvzf darshan-3.1.4.tar.gz</p>
                          <p>cd darshan-3.1.4/darshan-runtime</p>
                          <p>./configure --with-mem-align=8
                            --with-log-path=<log directory>
                            --prefix=<install directory>
                            --with-jobid-env=PBS_JOBID --disable-cuserid
                            CC=mpicc</p>
                          <p>make</p>
                          <p>make install</p>
                          <p><br>
                          </p>
                          <p>2. Create log directory, which successfully
                            create the directory </p>
                          <p><span style="text-decoration:underline"><a href="http://darshan-mk-log-dirs.pl/" target="_blank">darshan-mk-log-dirs.pl</a></span></p>
                          <p><br>
                          </p>
                          <p>3. Generate wrapper</p>
                          <p><span style="text-decoration:underline;color:rgb(228,175,10)"><a href="http://darshan-gen-cc.pl/" target="_blank">darshan-gen-cc.pl</a></span>
                            `which mpicc` --output mpicc.darshan</p>
                          <p><br>
                          </p>
                          <p>4. Compile application (mpi-io-test.c)
                            using mpicc.darshan</p>
                          <p>mpicc.darshan mpi-io-test.c -o mpi-io-test</p>
                          <p><br>
                          </p>
                          <p>After step 4, the following issue happened:</p>
                          <p>initthread.c:(.text+0xc59d): undefined
                            reference to `__wrap_fprintf'</p>
                          <p>initthread.c:(.text+0xc5bd): undefined
                            reference to `__wrap_fputs'</p>
                          <p>/global/homes/t/user/software_<wbr>install/mpich3.3a-static/lib/l<wbr>ibmpi.a(lib_libmpi_la-finalize<wbr>.o):
                            In function `MPIR_Add_finalize':</p>
                          <p>finalize.c:(.text+0x226e): undefined
                            reference to `__wrap_fputs'</p>
                          <p>finalize.c:(.text+0x227a): undefined
                            reference to `__wrap_fflush'</p>
                          <p>finalize.c:(.text+0x2286): undefined
                            reference to `__wrap_fflush'</p>
                          <p>/global/homes/t/user/software_<wbr>install/mpich3.3a-static/lib/l<wbr>ibmpi.a(lib_libmpi_la-util.o):
                            In function `MPIDI_OFI_control_handler':</p>
                          <p>util.c:(.text+0x17bc4): undefined reference
                            to `__wrap_fprintf'</p>
                          <p>/global/homes/t/user/software_<wbr>install/mpich3.3a-static/lib/l<wbr>ibmpi.a(lib_libmpi_la-ch4_glob<wbr>als.o):
                            In function `MPID_Abort':</p>
                          <p>ch4_globals.c:(.text+0x38b): undefined
                            reference to `__wrap_fputs'</p>
                          <p>ch4_globals.c:(.text+0x397): undefined
                            reference to `__wrap_fflush'</p>
                          <p>ch4_globals.c:(.text+0x3a3): undefined
                            reference to `__wrap_fflush'</p>
                          <p>/tmp/ccyntLoA.o: In function `main':</p>
                          <p>mpi-io-test.c:(.text+0x5ae): undefined
                            reference to `__wrap_printf'</p>
                          <p>mpi-io-test.c:(.text+0x819): undefined
                            reference to `__wrap_printf'</p>
                          <p>/global/homes/t/user/software_<wbr>install/mpich3.3a-static/lib/l<wbr>ibmpi.a(lib_libmpi_la-contexti<wbr>d.o):
                            In function `MPIR_Free_contextid':</p>
                          <p>contextid.c:(.text+0x1c3f): undefined
                            reference to `__wrap_fputs'</p>
                          <p>contextid.c:(.text+0x1c4b): undefined
                            reference to `__wrap_fflush'</p>
                          <p>contextid.c:(.text+0x1c57): undefined
                            reference to `__wrap_fflush'</p>
                          <p><br>
                          </p>
                          <p><br>
                          </p>
                          <p>mpicc.darshan -show mpi-io-test.c -o
                            mpi-io-test gave:</p>
                          <p><br>
                          </p>
                          <p>gcc -L/usr/lib64/slurmpmi
                            -L/global/common/cori/software<wbr>/libfabric/1.4.1/gnu/lib
                            mpi-io-test.c -o mpi-io-test
                            -L/global/homes/t/user/softwar<wbr>e_install/darshan/lib
                            -ldarshan -lz -Wl,@/global/homes/t/user/soft<wbr>ware_install/darshan/share/ld-<wbr>opts/darshan-base-ld-opts
                            -I/global/homes/t/user/softwar<wbr>e_install/mpich3.3a-static/<wbr>include
                            -L/global/homes/t/user/softwar<wbr>e_install/mpich3.3a-static/lib
                            -lmpi -lpmi -lpmi -lpthread -lfabric -lrt
                            -lpmi -L/global/homes/t/user/softwar<wbr>e_install/darshan/lib
                            -Wl,--start-group -ldarshan -ldarshan-stubs
                            -Wl,--end-group -lz -lrt -lpthread</p>
                          <p><br>
                          </p>
                          <p>The strange thing is when I do </p>
                          <p>nm libdarshan.a, it did show the functions
                            (e.g. __wrap_printf) were linked to </p>
                          <p>libdarshan.a. </p>
                          <p><br>
                          </p>
                          <p><b>0000000000002bf0 T __wrap__IO_getc</b></p>
                          <p><b>0000000000002d80 T __wrap__IO_putc</b></p>
                          <p><b>00000000000030b0 T
                              __wrap___isoc99_fscanf</b></p>
                          <p><b>0000000000001990 T __wrap_fclose</b></p>
                          <p><b>00000000000011d0 T __wrap_fdopen</b></p>
                          <p><b>0000000000001810 T __wrap_fflush</b></p>
                          <p><b>0000000000002a60 T __wrap_fgetc</b></p>
                          <p><b>0000000000003690 T __wrap_fgets</b></p>
                          <p><b>0000000000000db0 T __wrap_fopen</b></p>
                          <p><b>0000000000000fc0 T __wrap_fopen64</b></p>
                          <p><b>00000000000026b0 T __wrap_fprintf</b></p>
                          <p><b>0000000000001c80 T __wrap_fputc</b></p>
                          <p><b>0000000000001fa0 T __wrap_fputs</b></p>
                          <p><b>00000000000028d0 T __wrap_fread</b></p>
                          <p><b>00000000000013d0 T __wrap_freopen</b></p>
                          <p><b>00000000000015f0 T __wrap_freopen64</b></p>
                          <p><b>00000000000032d0 T __wrap_fscanf</b></p>
                          <p><b>0000000000003920 T __wrap_fseek</b></p>
                          <p><b>0000000000003a30 T __wrap_fseeko</b></p>
                          <p><b>0000000000003b40 T __wrap_fseeko64</b></p>
                          <p><b>0000000000003c50 T __wrap_fsetpos</b></p>
                          <p><br>
                          </p>
                          <p>Thanks,</p>
                          <p>Teng</p>
                        </div>
                        <br>
                        <fieldset class="m_-8621407278620756273m_6259078016213707887mimeAttachmentHeader"></fieldset>
                        <br>
                      </div>
                    </div>
                    <pre>______________________________<wbr>_________________
Darshan-users mailing list
<a class="m_-8621407278620756273m_6259078016213707887moz-txt-link-abbreviated" href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.go<wbr>v</a>
<a class="m_-8621407278620756273m_6259078016213707887moz-txt-link-freetext" href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" target="_blank">https://lists.mcs.anl.gov/mail<wbr>man/listinfo/darshan-users</a>
</pre>
                  </blockquote>
                  <br>
                </div>
                <br>
                ______________________________<wbr>_________________<br>
                Darshan-users mailing list<br>
                <a href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.go<wbr>v</a><br>
                <a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" rel="noreferrer" target="_blank">https://lists.mcs.anl.gov/mail<wbr>man/listinfo/darshan-users</a><br>
                <br>
              </blockquote>
            </div>
            <br>
          </div>
        </blockquote>
        <br>
        <br>
        <fieldset class="m_-8621407278620756273mimeAttachmentHeader"></fieldset>
        <br>
        <pre>______________________________<wbr>_________________
Darshan-users mailing list
<a class="m_-8621407278620756273moz-txt-link-abbreviated" href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.<wbr>gov</a>
<a class="m_-8621407278620756273moz-txt-link-freetext" href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" target="_blank">https://lists.mcs.anl.gov/<wbr>mailman/listinfo/darshan-users</a>
</pre>
      </blockquote>
      <br>
      <br>
      <fieldset class="m_-8621407278620756273mimeAttachmentHeader"></fieldset>
      <br>
      <pre>______________________________<wbr>_________________
Darshan-users mailing list
<a class="m_-8621407278620756273moz-txt-link-abbreviated" href="mailto:Darshan-users@lists.mcs.anl.gov" target="_blank">Darshan-users@lists.mcs.anl.<wbr>gov</a>
<a class="m_-8621407278620756273moz-txt-link-freetext" href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" target="_blank">https://lists.mcs.anl.gov/<wbr>mailman/listinfo/darshan-users</a>
</pre>
    </blockquote>
    <p><br>
    </p>
  </div></div></div>

<br>______________________________<wbr>_________________<br>
Darshan-users mailing list<br>
<a href="mailto:Darshan-users@lists.mcs.anl.gov">Darshan-users@lists.mcs.anl.<wbr>gov</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users" rel="noreferrer" target="_blank">https://lists.mcs.anl.gov/<wbr>mailman/listinfo/darshan-users</a><br>
<br></blockquote></div><br></div>