<div dir="ltr"><div><div><div><div>PETSc's design of looking for RC files in the users home directory really sucks. I've complained about this before a few years ago and am going to again.<br><br></div>A perfectly reasonable apps person had a .petscrc file in his home directory with at "%-pc_type_hypre" in it. This gave an error but he could not figure out where PETSc got this thing. (The error message was garbled for some reason, which slowed things down. As soon as I saw "%-pc_type_hypre" I knew what the problem was.) As you can see below he spent a day doing this.<br><br></div>I spent a day with another apps person on this same project a few years ago with this same problem. This is an error prone construct and it does not show up until you have used PETSc for a few years and have forgotten that you have a .petscrc file in your home directory. Very bad.<br><br></div>Also, this code specified that RC file name as "petsc.rc". It looks like PETSc is still picking up a .petscrc file anyway!!! At the very least we should scrub ".petscrc" if the user supplies another name.<br><br></div><div>I think we should dump this design and stop looking at home directories and just tell users to change, as we always do when we improve the design.<br></div><div><br></div><div>Thanks,<br></div>Mark<br><div><div><br><div><div><div><div><div><div><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Yoon, Eisung</b> <span dir="ltr"><<a href="mailto:yoone@rpi.edu">yoone@rpi.edu</a>></span><br>Date: Mon, Apr 13, 2015 at 11:00 PM<br>Subject: RE: related to compiling your source code<br>To: Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>>, Robert Hager <<a href="mailto:rhager@pppl.gov">rhager@pppl.gov</a>><br>Cc: Choong-Seock Chang <<a href="mailto:cschang@pppl.gov">cschang@pppl.gov</a>>, Mark Shephard <<a href="mailto:shephard@rpi.edu">shephard@rpi.edu</a>>, Seung-Hoe Ku <<a href="mailto:sku@pppl.gov">sku@pppl.gov</a>><br><br><br>
<div><p>--------<br>
WARNING: At least one of the links in the message below goes to an IP address (e.g.10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: <a href="https://commons.lbl.gov/x/_591B" target="_blank">https://commons.lbl.gov/x/_591B</a><br>
--------</p>
<div style="direction:ltr;font-family:Tahoma;color:rgb(0,0,0);font-size:10pt">
<div>Dear Mark and Robert,</div>
<div><br>
</div>
Wow! Thank you so much for all your comments and helps. After I spent a whole day, I am about to give up resolving this issue.
<div><br>
</div>
<div>I confirmed that the file which has %-pc_type hypre is located at my home directory! But the file name caused the problem is not petsc.rc but .petscrc, which I guess I copied a long time ago. The source code directory where the XGC execution file is located
has a petsc.rc which does not contain %-pc_type hypre and does not have .petscrc file in the directory. </div>
<div><br>
</div>
<div>In summary, three directories were involved for running XGC; A source code directory where the XGC execution file is located, working directory where job is submitted, and just my home directory. And the problem was from my home directory.</div>
<div><br>
</div>
<div>In addition, it is quite interesting that petscinitialize subroutine passes "./petsc.rc" in the XGC source code, while petsc tried to find a DEFAULT file ".petscrc" in my HOME directory!!!</div>
<div>
<div><br>
</div>
<div>I currently have removed .petscrc and submitted the job to see if XGC runs. </div>
<div><br>
</div>
<div>Best,</div>
<div>Eisung Yoon</div>
<div><br>
<div style="font-family:Times New Roman;color:rgb(0,0,0);font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" color="#000000" size="2"><b>From:</b> Mark Adams [<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>]<br>
<b>Sent:</b> Monday, April 13, 2015 10:19 PM<br>
<b>To:</b> Robert Hager<br>
<b>Cc:</b> Yoon, Eisung; Choong-Seock Chang; Mark Shephard; Seung-Hoe Ku<div><div class="h5"><br>
<b>Subject:</b> Re: related to compiling your source code<br>
</div></div></font><br>
</div><div><div class="h5">
<div></div>
<div>
<div dir="ltr">
<div>
<div>
<div>Good try Robert :)<br>
<br>
</div>
I'll bet Eisung has a petsc.rc file in his home directory. Let me know. I will use this as another data point to support my opinion that looking in your home directory is a bad idea.<br>
<br>
</div>
BTW, Seung-Hoe (cc'ed) and I had this same problem a few years ago and it took us hours to figure it out,<br>
<br>
</div>
Mark<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Apr 13, 2015 at 9:21 PM, Robert Hager <span dir="ltr">
<<a href="mailto:rhager@pppl.gov" target="_blank">rhager@pppl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word"><span>
<blockquote type="cite"><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium">
<div style="word-wrap:break-word">
<div style="direction:ltr;font-family:Tahoma;color:rgb(0,0,0);font-size:10pt">
<div>It seems Petsc is looking at a certain directory, but cannot check where it is.</div>
</div>
</div>
</span></blockquote>
<div><br>
</div>
</span>
<div>This may be a clue. I always copy the executable to my run directory and call something like</div>
<div><br>
</div>
<div>aprun ... ./xgca</div>
<div><br>
</div>
<div>In one of your earlier e-mails, I saw that you call</div>
<div><br>
</div>
<div>aprun ... {PATH_TO_XGCa_SOURCE}/xgca</div>
<div><br>
</div>
<div>If PETSc looks for petsc.rc in the directory of the executable, it will try to read a very old petsc.rc file that certainly does not work. Could you try with the executable to your run directory?</div>
<div><br>
</div>
<div>Best</div>
<div><br>
</div>
<div>Robert</div>
<br>
<div>
<div>
<div>
<div>On Apr 13, 2015, at 8:58 PM, Yoon, Eisung wrote:</div>
<br>
</div>
</div>
<blockquote type="cite"><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium">
<div style="word-wrap:break-word">
<div>
<div>
<div style="direction:ltr;font-family:Tahoma;color:rgb(0,0,0);font-size:10pt">
I attach the requested files.
<div><br>
</div>
<div>I tried PETSc and petsc.rc file in the XGC1 example suggested by Mark as well as the original input files in xgc_chang-hinton_test.tar. Also I checked language options which are same with yours and tried sed command, but all failed with almost same messages. </div>
<div><br>
</div>
<div>There were rarely interesting error messages, which can be a clue to resolve this issue. The error messages showed "Unknown statement in options file: (%-pc_type hypre )" even though my petsc.rc doesn't have that line. I checked petsc.rc files in XGC source
file directory as well as working(running) directory, but that line doesn't exist. Also the default .petscrc doesn't exist in both directories. It seems Petsc is looking at a certain directory, but cannot check where it is.</div>
<div><br>
</div>
<div><span style="font-size:10pt">Best,</span></div>
<div>Eisung Yoon</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<div style="font-family:"Times New Roman";color:rgb(0,0,0);font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" color="#000000" size="2"><b>From:</b><span> </span>Robert Hager [<a href="mailto:rhager@pppl.gov" target="_blank">rhager@pppl.gov</a>]<br>
<b>Sent:</b><span> </span>Monday, April 13, 2015 5:31 PM<br>
<b>To:</b><span> </span>Yoon, Eisung<br>
<b>Cc:</b><span> </span>Choong-Seock Chang; Mark Adams; Mark Shephard<br>
<b>Subject:</b><span> </span>Re: related to compiling your source code<br>
</font><br>
</div>
<div></div>
<div>That looks ok.
<div><br>
</div>
<div>I unpacked the tar-file I gave you and ran a diff with the petsc.rc that is still working for me and found that they are identical.</div>
<div><br>
</div>
<div>Did you edit any of the files (possibly in a Microsoft environment)? Or maybe your shell misinterprets characters. Did you specify any language in your shell setup?</div>
<div><br>
</div>
<div>In case something added any control characters to the petsc.rc file, you can run</div>
<div><br>
</div>
<div>
<div style="margin:0px;font:11px Menlo;color:rgb(211,45,38)">
<span style="color:rgb(0,0,0)">sed -e<span> </span></span>'s/[^[:print:]]//g'</div>
<div><br>
</div>
<div>to remove them.</div>
<div><br>
</div>
<div>My language settings are</div>
<div><br>
</div>
<blockquote style="margin:0px 0px 0px 40px;border-style:none;border-width:initial;border-color:initial;padding:0px">
<div>
<div><b>rhager@edison02:~/w/xgca_chang-hinton_test3> locale</b></div>
</div>
<div>
<div>LANG=</div>
</div>
<div>
<div>LC_CTYPE="POSIX"</div>
</div>
<div>
<div>LC_NUMERIC="POSIX"</div>
</div>
<div>
<div>LC_TIME="POSIX"</div>
</div>
<div>
<div>LC_COLLATE="POSIX"</div>
</div>
<div>
<div>LC_MONETARY="POSIX"</div>
</div>
<div>
<div>LC_MESSAGES="POSIX"</div>
</div>
<div>
<div>LC_PAPER="POSIX"</div>
</div>
<div>
<div>LC_NAME="POSIX"</div>
</div>
<div>
<div>LC_ADDRESS="POSIX"</div>
</div>
<div>
<div>LC_TELEPHONE="POSIX"</div>
</div>
<div>
<div>LC_MEASUREMENT="POSIX"</div>
</div>
<div>
<div>LC_IDENTIFICATION="POSIX"</div>
</div>
<div>
<div>LC_ALL=</div>
</div>
</blockquote>
<div><br>
</div>
<div>
<div> Could you send your makefile, <a href="http://defs.mk" target="_blank">defs.mk</a> and
<a href="http://rules.mk" target="_blank">rules.mk</a> (possibly <a href="http://rules_edison.mk" target="_blank">
rules_edison.mk</a>) anyway, please?</div>
<div><br>
</div>
<div>Best regards</div>
<div><br>
</div>
<div>Robert<br>
<div><br>
</div>
<div><br>
</div>
<div><br>
<div>
<div>On Apr 13, 2015, at 5:01 PM, Yoon, Eisung wrote:</div>
<br>
<blockquote type="cite">
<div style="word-wrap:break-word">
<div style="direction:ltr;font-family:Tahoma;color:rgb(0,0,0);font-size:10pt">
Hi Robert,
<div><br>
</div>
<div>I added below to .cshrc.ext as you recommended</div>
<div><br>
</div>
<div>
<div> module load cray-petsc</div>
<div> module load cray-hdf5-parallel</div>
<div> module load pspline</div>
<div> module load adios/1.6.0</div>
</div>
<div><br>
</div>
<div>and got</div>
<div><br>
</div>
<div>
<div>Currently Loaded Modulefiles:</div>
<div> 1) modules/<a href="http://3.2.10.2" target="_blank">3.2.10.2</a> 7) intel/<a href="http://15.0.1.133" target="_blank">15.0.1.133</a> 13) gni-headers/3.0-1.0502.9684.5.2.ari 19) PrgEnv-intel/5.2.40
25) altd/2.0 31) adios/1.6.0</div>
<div> 2) nsg/1.2.0 8) cray-libsci/13.0.1 14) xpmem/0.1-2.0502.55507.3.2.ari 20) craype-ivybridge 26) darshan/2.3.0</div>
<div> 3) eswrap/1.1.0-1.020200.1130.0 9) udreg/2.3.2-1.0502.9275.1.12.ari 15) dvs/2.5_0.9.0-1.0502.1873.1.145.ari 21) cray-shmem/7.1.1 27) usg-default-modules/1.1</div>
<div> 4) switch/1.0-1.0502.54233.2.96.ari 10) ugni/5.0-1.0502.9685.4.24.ari 16) alps/5.2.1-2.0502.9041.11.6.ari 22) cray-mpich/7.1.1 28) cray-petsc/<a href="http://3.5.2.1" target="_blank">3.5.2.1</a></div>
<div> 5) craype-network-aries 11) pmi/5.0.6-1.0000.10439.140.2.ari 17) rca/1.0.0-2.0502.53711.3.127.ari 23) torque/5.0.1 29) cray-hdf5-parallel/1.8.13</div>
<div> 6) craype/2.2.1 12) dmapp/7.0.1-1.0502.9501.5.219.ari 18) atp/1.7.5 24) moab/8.0.1-2014110616-5c7a394-sles11 30) pspline/nersc1.0</div>
</div>
<div><br>
</div>
<div>I copied Makefile.edison to Makefile, and had no problem with compiling and linking. I will try to figure out with the petsc.rc file. Thank you!</div>
<div><br>
</div>
<div>Best,</div>
<div>Eisung Yoon</div>
<div>
<div style="font-family:"Times New Roman";color:rgb(0,0,0);font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" color="#000000" size="2"><b>From:</b><span> </span>Robert Hager [<a href="mailto:rhager@pppl.gov" target="_blank">rhager@pppl.gov</a>]<br>
<b>Sent:</b><span> </span>Monday, April 13, 2015 4:55 PM<br>
<b>To:</b><span> </span>Choong-Seock Chang<br>
<b>Cc:</b><span> </span>Yoon, Eisung; Mark Adams; Mark Shephard<br>
<b>Subject:</b><span> </span>Re: related to compiling your source code<br>
</font><br>
</div>
<div></div>
<div>Hi Eisung,
<div><br>
</div>
<div>I used this file with XGCa on Edison today. Which modules do you use and which set of makefiles?</div>
<div><br>
</div>
<div>Best</div>
<div><br>
</div>
<div>Robert</div>
<div><br>
<div>
<div>On Apr 13, 2015, at 4:44 PM, Choong-Seock Chang wrote:</div>
<br>
<blockquote type="cite">
<div style="word-wrap:break-word">Please include Mark Adams in the PETSc related e-mails.
<div>He is in charge of PETSc in our project. He needs to be aware of all the conversations.<br>
<div>Thanks,</div>
<div>CS</div>
<div><br>
<div>
<div>On Apr 13, 2015, at 4:42 PM, Yoon, Eisung <<a href="mailto:yoone@rpi.edu" target="_blank">yoone@rpi.edu</a>> wrote:</div>
<br>
<blockquote type="cite">
<div style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="direction:ltr;font-family:Tahoma;font-size:10pt">Hi Robert,
<div><br>
</div>
<div>I tried to run XGC in Greene and Edison. Green still has a problem with PETSc. Even in edison, XGCa shows an error related to the petsc.rc file as below. Considering "invalid argument" in the message, I guess the petsc.rc included in the xgca_chang-hinton_test.tar
doesn't work. Unfortunately, the characters for the unknown option shown in the message is broken. Do you have working petsc.rc?</div>
<div><br>
</div>
<div>Thank you!</div>
<div>ES</div>
<div><br>
</div>
<div>
<div>(t_initf) Read in prof_inparam namelist from: input</div>
<div> PERF_SETOPTS: PAPI library not linked in. Request to enable PAPI ignored.</div>
<div> (t_initf) Using profile_disable= F profile_timer= 2</div>
<div> (t_initf) profile_depth_limit= 99999 profile_detail_limit= 1</div>
<div> (t_initf) profile_barrier= F profile_outpe_num= 1</div>
<div> (t_initf) profile_outpe_stride= 1 profile_single_file= F</div>
<div> (t_initf) profile_global_stats= T profile_papi_enable= F</div>
<div> call petsc_init</div>
<div>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div>
<div>[0]PETSC ERROR: Invalid argument</div>
<div>[0]PETSC ERROR: Unknown statement in options file: (�~A'^D)</div>
<div>[0]PETSC ERROR: See<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><span> </span>for trouble shooting.</div>
<div>[0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014</div>
<div>[0]PETSC ERROR: /global/u2/e/eyoon/branch/dev_rhager_esyoon/epsi/XGCa/xgca on a sandybridge named nid05677 by eyoon Mon Apr 13 13:31:32 2015</div>
<div>[0]PETSC ERROR: Configure options --known-mpi-int64_t=0 --known-bits-per-byte=8 --known-level1-dcache-assoc=0 --known-level1-dcache-linesize=32 --known-level1-dcache-size=32768 --known-memcmp-ok=1 --known-mpi-c-double-complex=1 --known-mpi-long-double=1
--known-mpi-shared-libraries=0 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8
--known-sizeof-void-p=8 --with-ar=ar --with-batch=1 --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=0 --with-dependencies=0 --with-fc=ftn --with-fortran-datatypes=0 --with-fortran-interfaces=0 --with-fortranlib-autodetect=0
--with-ranlib=ranlib --with-scalar-type=real --with-shared-ld=ar --with-etags=0 --with-dependencies=0 --with-dependencies=0 --with-mpi-dir=/opt/cray/mpt/7.0.0/gni/mpich2-intel/140 --with-superlu=1 --with-superlu-include=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/include
--with-superlu-lib=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/lib/libsuperlu.a --with-superlu_dist=1 --with-superlu_dist-include=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/include --with-superlu_dist-lib=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/lib/libsuperlu_dist.a
--with-parmetis=1 --with-parmetis-include=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/include --with-parmetis-lib=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/lib/libparmetis.a --with-metis=1 --with-metis-include=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/include
--with-metis-lib=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/lib/libmetis.a --with-ptscotch=1 --with-ptscotch-include=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/include --with-ptscotch-lib="-L/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/lib -lptscotch -lscotch
-lptscotcherr -lscotcherr" --with-scalapack=1 --with-scalapack-include=/opt/cray/libsci/13.0.0/INTEL/140/sandybridge/include --with-scalapack-lib="-L/opt/cray/libsci/13.0.0/INTEL/140/sandybridge/lib -lsci_intel_mpi_mp -lsci_intel_mp" --with-mumps=1 --with-mumps-include=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/include
--with-mumps-lib="-L/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/lib -lcmumps -ldmumps -lesmumps -lsmumps -lzmumps -lmumps_common -lptesmumps -lpord" --CFLAGS="-xavx -openmp -O3 " --CXXFLAGS="-xavx -openmp -O3 " --FFLAGS="-xavx -openmp -O3 " --LIBS=-lstdc++
--CXX_LINKER_FLAGS= --PETSC_ARCH=sandybridge --prefix=/opt/cray/petsc/<a href="http://3.5.2.1/real/INTEL/140/sandybridge" target="_blank">3.5.2.1/real/INTEL/140/sandybridge</a> --with-hypre=1 --with-hypre-include=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/include
--with-hypre-lib=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/lib/libHYPRE.a --with-sundials=1 --with-sundials-include=/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/include --with-sundials-lib="-L/opt/cray/tpsl/1.4.3/INTEL/140/sandybridge/lib -lsundials_cvode -lsundials_cvodes
-lsundials_ida -lsundials_idas -lsundials_kinsol -lsundials_nvecparallel -lsundials_nvecserial"</div>
<div>[0]PETSC ERROR: #1 PetscOptionsInsertFile() line 534 in /b/cray-petsc/.cray-build/INTEL/140/sandybridge/cray-petsc-base-dynamic/petsc-3.5.2/src/sys/objects/options.c</div>
<div>[0]PETSC ERROR: #2 PetscOptionsInsert() line 716 in /b/cray-petsc/.cray-build/INTEL/140/sandybridge/cray-petsc-base-dynamic/petsc-3.5.2/src/sys/objects/options.c</div>
<div>[0]PETSC ERROR: PetscInitialize:Creating options database</div>
<div>PETSC ERROR: Logging has not been enabled.</div>
<div>You might have forgotten to call PetscInitialize().</div>
<div>Rank 0 [Mon Apr 13 13:31:32 2015] [c5-3c1s11n1] application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0</div>
<div>forrtl: error (76): Abort trap signal</div>
<div>Image PC Routine Line Source</div>
<div>xgca 0000000003363F21 Unknown Unknown Unknown</div>
<div>xgca 0000000003362677 Unknown Unknown Unknown</div>
<div>xgca 000000000331A2F4 Unknown Unknown Unknown</div>
<div>xgca 000000000331A106 Unknown Unknown Unknown</div>
<div>xgca 00000000032AE434 Unknown Unknown Unknown</div>
<div>xgca 00000000032B53B1 Unknown Unknown Unknown</div>
<div>xgca 0000000002F64B60 Unknown Unknown Unknown</div>
<div>xgca 0000000002F64B1B Unknown Unknown Unknown</div>
<div>xgca 0000000003371B11 Unknown Unknown Unknown</div>
<div>xgca 0000000003131922 Unknown Unknown Unknown</div>
<div>xgca 0000000003100063 Unknown Unknown Unknown</div>
<div>xgca 00000000008BD7F0 Unknown Unknown Unknown</div>
<div>xgca 00000000008B2241 Unknown Unknown Unknown</div>
<div>xgca 00000000008C1B41 Unknown Unknown Unknown</div>
<div>xgca 000000000042554B perf_monitor_mp_p 1875 module.F90</div>
<div>xgca 000000000051F3BE MAIN__ 95 main.F90</div>
<div>xgca 0000000000405DEE Unknown Unknown Unknown</div>
<div>xgca 000000000336B6C1 Unknown Unknown Unknown</div>
<div>xgca 0000000000405CD1 Unknown Unknown Unknown</div>
<div>_pmiu_daemon(SIGCHLD): [NID 05677] [c5-3c1s11n1] [Mon Apr 13 13:31:32 2015] PE RANK 0 exit signal Aborted</div>
<div>[NID 05677] 2015-04-13 13:31:32 Apid 11750871: initiated application termination</div>
<div>Application 11750871 exit codes: 134</div>
<div>Application 11750871 exit signals: Killed</div>
<div>Application 11750871 resources: utime ~60s, stime ~12s, Rss ~29844, inblocks ~3174405, outblocks ~8270892</div>
<div style="font-family:"Times New Roman";font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" size="2"><b>From:</b><span> </span>Robert Hager [<a href="mailto:rhager@pppl.gov" target="_blank">rhager@pppl.gov</a>]<br>
<b>Sent:</b><span> </span>Monday, April 13, 2015 2:16 PM<br>
<b>To:</b><span> </span>Yoon, Eisung<br>
<b>Cc:</b><span> </span><a href="mailto:shephard@rpi.edu" target="_blank">shephard@rpi.edu</a>;<span> </span><a href="mailto:cschang@pppl.gov" target="_blank">cschang@pppl.gov</a><br>
<b>Subject:</b><span> </span>Re: related to compiling your source code<br>
</font><br>
</div>
<div></div>
<div>Hi Eisung,
<div><br>
</div>
<div>you can use the input in</div>
<div><br>
</div>
<div>/project/projectdirs/m499/rhager/xgca_chang-hinton_test.tar</div>
<div><br>
</div>
<div>Let me know if you have trouble reading the file.</div>
<div><br>
</div>
<div>Best regards</div>
<div><br>
</div>
<div>Robert</div>
<div><br>
<div>
<div>On Apr 13, 2015, at 1:46 PM, Yoon, Eisung wrote:</div>
<br>
<blockquote type="cite">
<div style="direction:ltr;font-family:Tahoma;font-size:10pt">Hi Robert,
<div><br>
</div>
<div>Thank you for the information and explanation. I attach a text file which contains issues of source code with TRIGRID and variable collision time.</div>
<div><br>
</div>
<div>I'm sorry for not telling you previously that I was compiling the source code in PPPL server. I've not ready to use XGC in Edison yet but I'm going to work it to be ready right now.</div>
<div><br>
</div>
<div>Could you send me an input file of XGCa for a collision test in Edison?</div>
<div><br>
</div>
<div>Thanks a lot!!!</div>
<div>ES</div>
<div><br>
</div>
<div><br>
<div style="font-family:"Times New Roman";font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" size="2"><b>From:</b><span> </span>Robert Hager [<a href="mailto:rhager@pppl.gov" target="_blank">rhager@pppl.gov</a>]<br>
<b>Sent:</b><span> </span>Monday, April 13, 2015 10:34 AM<br>
<b>To:</b><span> </span>Yoon, Eisung<br>
<b>Cc:</b><span> </span><a href="mailto:shephard@rpi.edu" target="_blank">shephard@rpi.edu</a>;<span> </span><a href="mailto:cschang@pppl.gov" target="_blank">cschang@pppl.gov</a><br>
<b>Subject:</b><span> </span>Re: related to compiling your source code<br>
</font><br>
</div>
<div></div>
<div>Hi Eisung,
<div><br>
</div>
<div>the TRIGRID directive should not cause any errors. Can I see the error message?</div>
<div><br>
</div>
<div>I looked at Makefile.edison in your branch. It looks fine. You might have to change
<a href="http://defs.mk" target="_blank">defs.mk</a> though. There is one include statement to import some PETSc variable definitions. Depending on whether you use PETSc 3.5 or 3.6, you have to use the first or the second line, respectively.</div>
<div><br>
</div>
<div>On Edison, I load the following modules in addition to the default:</div>
<div><br>
</div>
<div>
<div> module load cray-petsc</div>
<div> module load cray-hdf5-parallel</div>
<div> module load pspline</div>
</div>
<div><br>
</div>
<blockquote style="margin:0px 0px 0px 40px;border-style:none;border-width:initial;border-color:initial;padding:0px">
<div>The output of module list is</div>
<div><br>
</div>
<div>Currently Loaded Modulefiles:</div>
<div> 1) modules/<a href="http://3.2.10.2" target="_blank">3.2.10.2</a> 13) gni-headers/3.0-1.0502.9684.5.2.ari 25) cray-petsc/<a href="http://3.5.2.1" target="_blank">3.5.2.1</a></div>
<div> 2) nsg/1.2.0 14) xpmem/0.1-2.0502.55507.3.2.ari 26) cray-hdf5-parallel/1.8.13</div>
<div> 3) eswrap/1.1.0-1.020200.1130.0 15) dvs/2.5_0.9.0-1.0502.1873.1.145.ari 27) pspline/nersc1.0</div>
<div> 4) switch/1.0-1.0502.54233.2.96.ari 16) alps/5.2.1-2.0502.9041.11.6.ari 28) allineatools/5.0.1</div>
<div> 5) craype-network-aries 17) rca/1.0.0-2.0502.53711.3.127.ari 29) idl/8.2</div>
<div> 6) craype/2.2.1 18) atp/1.7.5 30) gv/3.7.3</div>
<div> 7) intel/<a href="http://15.0.1.133" target="_blank">15.0.1.133</a> 19) PrgEnv-intel/5.2.40 31) latex/2012</div>
<div> 8) cray-libsci/13.0.1 20) craype-ivybridge 32) altd/2.0</div>
<div> 9) udreg/2.3.2-1.0502.9275.1.12.ari 21) cray-shmem/7.1.1 33) darshan/2.3.0</div>
<div> 10) ugni/5.0-1.0502.9685.4.24.ari 22) cray-mpich/7.1.1 34) usg-default-modules/1.1</div>
<div> 11) pmi/5.0.6-1.0000.10439.140.2.ari 23) torque/5.0.1</div>
<div> 12) dmapp/7.0.1-1.0502.9501.5.219.ari 24) moab/8.0.1-2014110616-5c7a394-sles11</div>
</blockquote>
<div><br>
</div>
<div>Last time I tried, the code compiled with these settings. It also ran a couple of time steps. But there are still some bugs in the code. Making the collision time step variable is a bit complicated because the collision operation is usually run together
with all other sources like heating, etc. Therefore, the distribution function is evaluated only every sml_f_source_period time steps. If a collision operation is supposed to run at a different time step, f will not be available with the current code. However,
in order to test whether it is worth to pursue this approach, I wanted to implement variable collision time steps in the simplest possible way, i.e. sml_f_source_period=0 and all sources except the collision operation deactivated. The collision interval must
have an upper limit which I set to 10 time steps in my test. The interval for load-balancing should be a multiple of this upper limit in order to be efficient. If this approach helps to improve performance, we can think about how to implement variable collision
intervals in a cleaner way.</div>
<div><br>
</div>
<div>Let me know if you have any further problems.</div>
<div><br>
</div>
<div>Best</div>
<div><br>
</div>
<div>Robert</div>
<div><br>
</div>
<div><br>
<div>
<div>On Apr 12, 2015, at 2:38 PM, Yoon, Eisung wrote:</div>
<br>
<blockquote type="cite">
<div>
<div style="direction:ltr;font-family:Tahoma;font-size:10pt">Hi Robert,
<div><br>
</div>
<div>Thank you for the performance test data. I really appreciate your work.</div>
<div><br>
</div>
<div>As for variable collision time, <span style="font-size:10pt">I've made a branch "dev_rhager_esyoon" as a copy of your source code, "dev_rhager". I've read your modification for variable collision time in the XGCa folder.</span></div>
<div><br>
</div>
<div>In order to run the code, I currently have trouble with compiling the source code. It appears preprocessing directives -DTRIGRID causes the error. Could you send me your Makefile to see working compile options?</div>
<div><br>
</div>
<div>Thank you.</div>
<div><br>
</div>
<div>Best,</div>
<div>ES</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
<span style="font-family:Helvetica"><code_reading.txt></span></blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<span><<a href="http://defs.mk" target="_blank">defs.mk</a>></span><span><Makefile></span><span><<a href="http://rules.mk" target="_blank">rules.mk</a>></span></div>
</span></blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div></div></div>
</div>
</div>
</div>
</div>
</div><br></div></div></div></div></div></div></div></div></div>