<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 10, 2017 at 4:28 PM, Satish Balay <span dir="ltr"><<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Yup. Thats 4294967296*8bytes 34GB RAM - just for this one hash table?<br></blockquote><div><br></div><div>Humm, that is crazy large.</div><div><br></div><div>Treb: I would like to get the stack trace. Can you please send the whole output or grep on "[5]" (or any proc that fails) to get a stack trace. And can you tell me how many global equations you have here.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Satish<br>
<span><br>
On Tue, 10 Jan 2017, Kong, Fande wrote:<br>
<br>
> On Tue, Jan 10, 2017 at 1:47 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
><br>
> > We have a code that is failing on large scale runs on Edison at NERSC. We<br>
> > ran with a debug version and we got this error message that I assume is<br>
> > from a check that is done in debug. Is this a false positive? This is a 64<br>
> > bit integer build.<br>
> > Thanks,<br>
> > Mark<br>
> ><br>
> > [5]PETSC ERROR: --------------------- Error Message<br>
> > ------------------------------<wbr>------------------------------<wbr>--<br>
> > [5]PETSC ERROR: Argument out of range<br>
> > [5]PETSC ERROR: A really huge hash is being requested.. cannot process:<br>
> > 4294967296<br>
> ><br>
><br>
> Satish,<br>
><br>
> The number "4294967296" is larger than the maximum number <a href="tel:%282147321881" value="+12147321881" target="_blank">(2147321881</a>) you<br>
> put in the lookup.<br>
><br>
> I guess your fixes do not help this one.<br>
><br>
><br>
> Fande,<br>
><br>
><br>
><br>
><br>
> > [5]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a><br>
</span>> > <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_documentation_faq.html&d=DQMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=Jd9TVPRE-Ts5OmoJe38KFIo29VVtsFIoN1ABdechuuE&s=BdTH9phOP3ipZiM21mw_gZrxPjWTeVcpXw9n-y7GECo&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint<wbr>.com/v2/url?u=http-3A__www.<wbr>mcs.anl.gov_petsc_documentatio<wbr>n_faq.html&d=DQMFaQ&c=54IZrppP<wbr>QZKX9mLzcGdPfFD1hxrcB__<wbr>aEkJFOKJFd00&r=DUUt3SRGI0_JgtN<wbr>aS3udV68GRkgV4ts7XKfj2opmiCY&<wbr>m=Jd9TVPRE-Ts5OmoJe38KFIo29VVt<wbr>sFIoN1ABdechuuE&s=<wbr>BdTH9phOP3ipZiM21mw_gZrxPjWTeV<wbr>cpXw9n-y7GECo&e=</a>><br>
<div><div class="m_-108618658402229371h5">> > for trouble shooting.<br>
> > [5]PETSC ERROR: Petsc Development GIT revision: v3.7.4-2584-g7175242 GIT<br>
> > Date: 2016-12-21 13:19:16 -0600<br>
> > [5]PETSC ERROR: /tmp/targetExec on a arch-xc30-dbg64-intel named nid00008<br>
> > by u6338 Sat Jan 7 16:39:16 2017<br>
> > [5]PETSC ERROR: Configure options --COPTFLAGS="-O0 -g -no-ipo"<br>
> > --CXXOPTFLAGS="-O0 -g -no-ipo" --FOPTFLAGS="-O0 -g -ipo"<br>
> > --download-parmetis --download-metis --download-hypre --with-ssl=0<br>
> > --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC<br>
> > --with-cxxlib-autodetect=0 --with-debugging=1 --with-fc=ftn<br>
> > --with-fortranlib-autodetect=0 --with-shared-libraries=0 --with-x=0<br>
> > --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices<br>
> > --prefix=/global/homes/m/madam<wbr>s/petsc_install/petscv3.7-x[<wbr>16527]PETSC<br>
> > ERROR: ----<br>
> ><br>
> > using:<br>
> > PETSC_DIR=/global/homes/m/mada<wbr>ms/petsc_install/petscv3.7-<wbr>xc30-dbg64-intel/<br>
> ><br>
> > maybe go back to opt version?<br>
> ><br>
> ><br>
> > On 12/22/2016 03:22 PM, Mark Adams wrote:<br>
> ><br>
> >> You configure with the size of an integer explicitly, ie,<br>
> >> --with-64-bit-integers. I can't imagine there is anything bigger, but it<br>
> >> would not be hard to do.<br>
> >><br>
> >> On Thu, Dec 22, 2016 at 6:06 PM, David Trebotich <<a href="mailto:treb@hpcrdm.lbl.gov" target="_blank">treb@hpcrdm.lbl.gov</a>><br>
> >> wrote:<br>
> >><br>
> >> i just committed changes to SPMD that Brian needed for signed long long.<br>
> >>> Evidently petsc is now handling bigger matrices?<br>
> >>><br>
> >>><br>
> >>> On 12/22/2016 2:57 PM, Mark Adams wrote:<br>
> >>><br>
> >>> There are a few hits on stack exchange, google:<br>
> >>><br>
> >>> "error: expression must have class type"<br>
> >>><br>
> >>> I think intputT is a templated type and it is templated with a PetscInt.<br>
> >>> You are calling the linearSize() method on a primitive type. That is my<br>
> >>> best guess.<br>
> >>><br>
> >>> On Thu, Dec 22, 2016 at 3:33 PM, David Trebotich <<a href="mailto:treb@hpcrd.lbl.gov" target="_blank">treb@hpcrd.lbl.gov</a>><br>
> >>> wrote:<br>
> >>><br>
> >>> building with dbg looks same:<br>
> >>>><br>
> >>>> ../../../Chombo/lib/src/BaseTo<wbr>ols/SPMDI.H(23): error: expression must<br>
> >>>> have class type<br>
> >>>> return inputT.linearSize();<br>
> >>>> ^<br>
> >>>> detected during:<br>
> >>>> instantiation of "int linearSize(const T &) [with<br>
> >>>> T=PetscInt={PetscInt64={long long}}]" at line 128 of<br>
> >>>> "../../../Chombo/lib/src/EBToo<wbr>ls/MiniIVFABI.H"<br>
> >>>> instantiation of "int MiniIVFAB<T>::size(const Box &, const<br>
> >>>> Interval &) const [with T=PetscInt={PetscInt64={long long}}]" at line<br>
> >>>> 155<br>
> >>>> of "../../../Chombo/lib/src/EBToo<wbr>ls/BaseEBCellFABI.H"<br>
> >>>> instantiation of "int BaseEBCellFAB<T>::size(const Box &,<br>
> >>>> const Interval &) const [with T=PetscInt={PetscInt64={long long}}]" at<br>
> >>>> line<br>
> >>>> 149 of "../../../Chombo/lib/src/BoxTo<wbr>ols/BoxLayoutData.H"<br>
> >>>> instantiation of "int LDOperator<T>::size(const T &, const<br>
</div></div>> >>>> Box &, const Interval &) const [with T=BaseEBCellFAB<PetscInt={Pets<br>
> >>>> cInt64={long<br>
<div><div class="m_-108618658402229371h5">> >>>> long}}>]" at line 114 of "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>> implicit generation of "LDOperator<T>::LDOperator() [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114 of<br>
> >>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>> instantiation of class "LDOperator<T> [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114 of<br>
> >>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>> instantiation of "void LevelData<T>::copyTo(const Interval<br>
> >>>> &,<br>
> >>>> BoxLayoutData<T> &, const Interval &, const Copier &, const<br>
> >>>> LDOperator<T><br>
> >>>> &) const [with T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at<br>
> >>>> line 213 of "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelDataI.H"<br>
> >>>> instantiation of "void LevelData<T>::copyTo(const Interval<br>
> >>>> &,<br>
> >>>> BoxLayoutData<T> &, const Interval &) const [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 32 of<br>
> >>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelDataI.H"<br>
> >>>> instantiation of "LevelData<T>::LevelData() [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 41 of<br>
> >>>> "../srcLib/EBPetscSolver.cpp"<br>
> >>>><br>
> >>>> ../../../Chombo/lib/src/BaseTo<wbr>ols/SPMDI.H(35): error: expression must<br>
> >>>> have class type<br>
> >>>> inputT.linearOut(a_outBuf);<br>
> >>>> ^<br>
> >>>> detected during:<br>
> >>>> instantiation of "void linearOut(void *, const T &) [with<br>
> >>>> T=PetscInt={PetscInt64={long long}}]" at line 155 of<br>
> >>>> "../../../Chombo/lib/src/EBToo<wbr>ls/MiniIVFABI.H"<br>
> >>>> instantiation of "void MiniIVFAB<T>::linearOut(void *, const<br>
> >>>> Box &, const Interval &) const [with T=PetscInt={PetscInt64={long<br>
> >>>> long}}]"<br>
> >>>> at line 169 of "../../../Chombo/lib/src/EBToo<wbr>ls/BaseEBCellFABI.H"<br>
> >>>> instantiation of "void BaseEBCellFAB<T>::linearOut(vo<wbr>id *,<br>
> >>>> const Box &, const Interval &) const [with T=PetscInt={PetscInt64={long<br>
> >>>> long}}]" at line 154 of "../../../Chombo/lib/src/BoxTo<br>
> >>>> ols/BoxLayoutData.H"<br>
> >>>> instantiation of "void LDOperator<T>::linearOut(const T &,<br>
> >>>> void *, const Box &, const Interval &) const [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114 of<br>
> >>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>> implicit generation of "LDOperator<T>::LDOperator() [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114 of<br>
> >>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>> instantiation of class "LDOperator<T> [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114 of<br>
> >>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>> instantiation of "void LevelData<T>::copyTo(const Interval<br>
> >>>> &,<br>
> >>>> BoxLayoutData<T> &, const Interval &, const Copier &, const<br>
> >>>> LDOperator<T><br>
> >>>> &) const [with T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at<br>
> >>>> line 213 of "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelDataI.H"<br>
> >>>> instantiation of "void LevelData<T>::copyTo(const Interval<br>
> >>>> &,<br>
> >>>> BoxLayoutData<T> &, const Interval &) const [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 32 of<br>
> >>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelDataI.H"<br>
> >>>> instantiation of "LevelData<T>::LevelData() [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 41 of<br>
> >>>> "../srcLib/EBPetscSolver.cpp"<br>
> >>>><br>
> >>>> ../../../Chombo/lib/src/BaseTo<wbr>ols/SPMDI.H(29): error: expression must<br>
> >>>> have class type<br>
> >>>> a_outputT.linearIn(inBuf);<br>
> >>>> ^<br>
> >>>> detected during:<br>
> >>>> instantiation of "void linearIn(T &, const void *) [with<br>
> >>>> T=PetscInt={PetscInt64={long long}}]" at line 181 of<br>
> >>>> "../../../Chombo/lib/src/EBToo<wbr>ls/MiniIVFABI.H"<br>
> >>>> instantiation of "void MiniIVFAB<T>::linearIn(void *, const<br>
> >>>> Box &, const Interval &) [with T=PetscInt={PetscInt64={long long}}]" at<br>
> >>>> line 181 of "../../../Chombo/lib/src/EBToo<wbr>ls/BaseEBCellFABI.H"<br>
> >>>> instantiation of "void BaseEBCellFAB<T>::linearIn(voi<wbr>d *,<br>
> >>>> const Box &, const Interval &) [with T=PetscInt={PetscInt64={long<br>
> >>>> long}}]"<br>
> >>>> at line 159 of "../../../Chombo/lib/src/BoxTo<wbr>ols/BoxLayoutData.H"<br>
> >>>> instantiation of "void LDOperator<T>::linearIn(T &, void *,<br>
> >>>> const Box &, const Interval &) const [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long<br>
> >>>> long}}>]" at line 114 of "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>> implicit generation of "LDOperator<T>::LDOperator() [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114 of<br>
> >>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>> instantiation of class "LDOperator<T> [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114 of<br>
> >>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>> instantiation of "void LevelData<T>::copyTo(const Interval<br>
> >>>> &,<br>
> >>>> BoxLayoutData<T> &, const Interval &, const Copier &, const<br>
> >>>> LDOperator<T><br>
> >>>> &) const [with T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at<br>
> >>>> line 213 of "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelDataI.H"<br>
> >>>> instantiation of "void LevelData<T>::copyTo(const Interval<br>
> >>>> &,<br>
> >>>> BoxLayoutData<T> &, const Interval &) const [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 32 of<br>
> >>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelDataI.H"<br>
> >>>> instantiation of "LevelData<T>::LevelData() [with<br>
> >>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 41 of<br>
> >>>> "../srcLib/EBPetscSolver.cpp"<br>
> >>>><br>
> >>>> compilation aborted for ../srcLib/EBPetscSolver.cpp (code 2)<br>
> >>>> gmake[1]: *** [o/<a href="http://3d.Linux.64.CC.ftn.DEBUG.MP" rel="noreferrer" target="_blank">3d.Linux.64.CC.ftn.DEBUG.MP</a><br>
</div></div>> >>>> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__3d.Linux.64.CC.ftn.DEBUG.MP&d=DQMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=Jd9TVPRE-Ts5OmoJe38KFIo29VVtsFIoN1ABdechuuE&s=HAhvWGeoB7HMK7Clyrfh_x-R465qxP-CNXK4zqhrCuE&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint<wbr>.com/v2/url?u=http-3A__3d.<wbr>Linux.64.CC.ftn.DEBUG.MP&d=<wbr>DQMFaQ&c=54IZrppPQZKX9mLzcGdPf<wbr>FD1hxrcB__aEkJFOKJFd00&r=<wbr>DUUt3SRGI0_JgtNaS3udV68GRkgV4t<wbr>s7XKfj2opmiCY&m=Jd9TVPRE-Ts5Om<wbr>oJe38KFIo29VVtsFIoN1ABdechuuE&<wbr>s=HAhvWGeoB7HMK7Clyrfh_x-R465q<wbr>xP-CNXK4zqhrCuE&e=</a>><br>
<div><div class="m_-108618658402229371h5">> >>>> I.PETSC/EBPetscSolver.o]<br>
> >>>> Error 2<br>
> >>>><br>
> >>>><br>
> >>>> On 12/22/2016 09:42 AM, Mark Adams wrote:<br>
> >>>><br>
> >>>> I have build a debug version (opt64 --> dbg64). You might try that.<br>
> >>>>><br>
> >>>>> On Wed, Dec 21, 2016 at 9:46 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>>> On Wed, Dec 21, 2016 at 2:25 PM, David Trebotich <<a href="mailto:treb@hpcrd.lbl.gov" target="_blank">treb@hpcrd.lbl.gov</a>><br>
> >>>>>> wrote:<br>
> >>>>>><br>
> >>>>>> This wasn't happening until after you rebuilt petsc. So I am<br>
> >>>>>> wondering<br>
> >>>>>><br>
> >>>>>>> if you upgraded or changed something??<br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>><br>
> >>>>>> Humm, let me rebuild. This is on Edison I assume.<br>
> >>>>>><br>
> >>>>>> Try PETSC_DIR=/global/homes/m/mada<wbr>ms/petsc_install/petscv3.7-xc3<br>
> >>>>>> 0-opt64-intel<br>
> >>>>>> (again)<br>
> >>>>>><br>
> >>>>>> This is a new build. I would also do a deep clean on your end, of<br>
> >>>>>> course.<br>
> >>>>>><br>
> >>>>>> I see a DCH_USE_COMPLEX ...<br>
> >>>>>><br>
> >>>>>> Also, I would try a debug build just to get some data. This is some<br>
> >>>>>> strange problem of something is out of synch.<br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>> On 12/19/2016 05:05 PM, Mark Adams wrote:<br>
> >>>>>>><br>
> >>>>>>> On Mon, Dec 19, 2016 at 6:35 PM, David Trebotich <<a href="mailto:treb@hpcrd.lbl.gov" target="_blank">treb@hpcrd.lbl.gov</a><br>
> >>>>>>> ><br>
> >>>>>>><br>
> >>>>>>>> wrote:<br>
> >>>>>>>><br>
> >>>>>>>> I am sorry, you mean I need to add that to environment variables?<br>
> >>>>>>>><br>
> >>>>>>>><br>
> >>>>>>>>><br>
> >>>>>>>>><br>
> >>>>>>>> The PETSC_ARCH is hardwired for the size of integers so you would<br>
> >>>>>>>> use<br>
> >>>>>>>> PETSC_ARCH=arch-xc30-opt-intel to get 32 bit ints (I don't know of<br>
> >>>>>>>> this<br>
> >>>>>>>> installation ever being used). But now that I think about it you<br>
> >>>>>>>> always<br>
> >>>>>>>> use<br>
> >>>>>>>> 64 bit ints and you need to.<br>
> >>>>>>>><br>
> >>>>>>>> So what is going on here ... it looks like an object has been<br>
> >>>>>>>> templated<br>
> >>>>>>>> on<br>
> >>>>>>>> a PetscInt and is getting mixed up. I don't think you ever use 32<br>
> >>>>>>>> bit<br>
> >>>>>>>> ints<br>
> >>>>>>>> in PETSc so it looks like you regressed here... I don't have a lot<br>
> >>>>>>>> of<br>
> >>>>>>>> experience with templates. I'd ask Brian or Dan.<br>
> >>>>>>>><br>
> >>>>>>>><br>
> >>>>>>>><br>
> >>>>>>>><br>
> >>>>>>>> On 12/19/2016 03:34 PM, Mark Adams wrote:<br>
> >>>>>>>>><br>
> >>>>>>>>> There is a: arch-xc30-opt-intel<br>
> >>>>>>>>><br>
> >>>>>>>>><br>
> >>>>>>>>>> On Mon, Dec 19, 2016 at 6:25 PM, David Trebotich <<br>
> >>>>>>>>>> <a href="mailto:treb@hpcrd.lbl.gov" target="_blank">treb@hpcrd.lbl.gov</a>><br>
> >>>>>>>>>> wrote:<br>
> >>>>>>>>>><br>
> >>>>>>>>>> Hey Mark-<br>
> >>>>>>>>>><br>
> >>>>>>>>>> We have had this issue recently and am wondering if you know what<br>
> >>>>>>>>>><br>
> >>>>>>>>>>> the<br>
> >>>>>>>>>>> problem is. Maybe an int vs. PetscInt vs. PetscInt64 kind of<br>
> >>>>>>>>>>> thing?<br>
> >>>>>>>>>>><br>
> >>>>>>>>>>> CC -O2 -fast -no-ipo -std=c++11 -O3 -m64<br>
> >>>>>>>>>>> -qopt-multi-version-aggressive<br>
> >>>>>>>>>>> -restrict -m64 -std=c++11 -Wno-unknown-pragmas -diag-disable 3180<br>
> >>>>>>>>>>> -DCH_SPACEDIM=3 -DCH_Linux -DNDEBUG -DCH_MPI<br>
> >>>>>>>>>>> -DMPICH_SKIP_MPICXX<br>
> >>>>>>>>>>> -ULAM_WANT_MPI2CPP -DMPI_NO_CPPBIND -DCH_USE_COMPLEX<br>
> >>>>>>>>>>> -DCH_USE_EB<br>
> >>>>>>>>>>> -DCH_USE_MEMORY_TRACKING -DCH_USE_64 -DCH_USE_DOUBLE<br>
> >>>>>>>>>>> -DCH_USE_HDF5<br>
> >>>>>>>>>>> -I/opt/cray/hdf5-parallel/1.8.<wbr>16/INTEL/15.0/include -I.<br>
> >>>>>>>>>>> -I../srcBCG<br>
> >>>>>>>>>>> -I../srcLib -I../srcAdvectDiffuseCrunch_BC<wbr>G<br>
> >>>>>>>>>>> -I../srcCrunchFlow_BCG<br>
> >>>>>>>>>>> -I../srcCrunchFlow_BCG/<wbr>modules -DCH_USE_PETSC<br>
> >>>>>>>>>>> -I/global/homes/m/madams/petsc<wbr>_install/petscv3.6-xc30-opt64-<br>
> >>>>>>>>>>> intel/include<br>
> >>>>>>>>>>> -I/global/homes/m/madams/petsc<wbr>_install/petscv3.6-xc30-opt64-<br>
> >>>>>>>>>>> intel/include<br>
> >>>>>>>>>>> -I/global/homes/m/madams/petsc<wbr>_install/petscv3.6-xc30-opt64-<br>
> >>>>>>>>>>> intel/include<br>
> >>>>>>>>>>> -DCH_FORT_UNDERSCORE -I../../../Chombo/lib/src/EBAM<br>
> >>>>>>>>>>> RTimeDependent<br>
> >>>>>>>>>>> -I../../../Chombo/lib/src/EBAM<wbr>RElliptic<br>
> >>>>>>>>>>> -I../../../Chombo/lib/src/EBAM<br>
> >>>>>>>>>>> RTools<br>
> >>>>>>>>>>> -I../../../Chombo/lib/src/Work<wbr>shop<br>
> >>>>>>>>>>> -I../../../Chombo/lib/src/EBTo<br>
> >>>>>>>>>>> ols<br>
> >>>>>>>>>>> -I../../../Chombo/lib/src/AMRT<wbr>imeDependent<br>
> >>>>>>>>>>> -I../../../Chombo/lib/src/AMRE<wbr>lliptic<br>
> >>>>>>>>>>> -I../../../Chombo/lib/src/AMRT<wbr>ools<br>
> >>>>>>>>>>> -I../../../Chombo/lib/src/BoxT<br>
> >>>>>>>>>>> ools<br>
> >>>>>>>>>>> -I../../../Chombo/lib/src/Base<wbr>Tools -DCH_LANG_CC -c<br>
> >>>>>>>>>>> ../srcLib/EBPetscSolver.cpp -o o/3d.Linux.64.CC.ftn.OPTHIGH.M<br>
> >>>>>>>>>>> PI.PETSC/EBPetscSolver.o<br>
> >>>>>>>>>>> ../../../Chombo/lib/src/BaseTo<wbr>ols/SPMDI.H(23): error: expression<br>
> >>>>>>>>>>> must<br>
> >>>>>>>>>>> have class type<br>
> >>>>>>>>>>> return inputT.linearSize();<br>
> >>>>>>>>>>> ^<br>
> >>>>>>>>>>> detected during:<br>
> >>>>>>>>>>> instantiation of "int linearSize(const T &) [with<br>
> >>>>>>>>>>> T=PetscInt={PetscInt64={long long}}]" at line 128 of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/EBToo<wbr>ls/MiniIVFABI.H"<br>
> >>>>>>>>>>> instantiation of "int MiniIVFAB<T>::size(const Box &,<br>
> >>>>>>>>>>> const<br>
> >>>>>>>>>>> Interval &) const [with T=PetscInt={PetscInt64={long long}}]" at<br>
> >>>>>>>>>>> line<br>
> >>>>>>>>>>> 155<br>
> >>>>>>>>>>> of "../../../Chombo/lib/src/EBToo<wbr>ls/BaseEBCellFABI.H"<br>
> >>>>>>>>>>> instantiation of "int BaseEBCellFAB<T>::size(const<br>
> >>>>>>>>>>> Box<br>
> >>>>>>>>>>> &,<br>
> >>>>>>>>>>> const Interval &) const [with T=PetscInt={PetscInt64={long<br>
> >>>>>>>>>>> long}}]" at<br>
> >>>>>>>>>>> line<br>
> >>>>>>>>>>> 149 of "../../../Chombo/lib/src/BoxTo<wbr>ols/BoxLayoutData.H"<br>
> >>>>>>>>>>> instantiation of "int LDOperator<T>::size(const T &,<br>
> >>>>>>>>>>> const<br>
> >>>>>>>>>>> Box<br>
> >>>>>>>>>>> &, const Interval &) const [with T=BaseEBCellFAB<PetscInt={Pets<br>
> >>>>>>>>>>> cInt64={long<br>
> >>>>>>>>>>> long}}>]" at line 114 of "../../../Chombo/lib/src/BoxTo<br>
> >>>>>>>>>>> ols/LevelData.H"<br>
> >>>>>>>>>>> implicit generation of "LDOperator<T>::LDOperator()<br>
> >>>>>>>>>>> [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>>>>>>>>> instantiation of class "LDOperator<T> [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>>>>>>>>> instantiation of "void LevelData<T>::copyTo(const<br>
> >>>>>>>>>>> Interval &,<br>
> >>>>>>>>>>> BoxLayoutData<T> &, const Interval &, const Copier &, const<br>
> >>>>>>>>>>> LDOperator<T><br>
> >>>>>>>>>>> &) const [with T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long<br>
> >>>>>>>>>>> long}}>]"<br>
> >>>>>>>>>>> at<br>
> >>>>>>>>>>> line 213 of "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelDataI.H"<br>
> >>>>>>>>>>> instantiation of "void LevelData<T>::copyTo(const<br>
> >>>>>>>>>>> Interval &,<br>
> >>>>>>>>>>> BoxLayoutData<T> &, const Interval &) const [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 32<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelDataI.H"<br>
> >>>>>>>>>>> instantiation of "LevelData<T>::LevelData() [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 41<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../srcLib/EBPetscSolver.cpp"<br>
> >>>>>>>>>>><br>
> >>>>>>>>>>> ../../../Chombo/lib/src/BaseTo<wbr>ols/SPMDI.H(35): error: expression<br>
> >>>>>>>>>>> must<br>
> >>>>>>>>>>> have class type<br>
> >>>>>>>>>>> inputT.linearOut(a_outBuf);<br>
> >>>>>>>>>>> ^<br>
> >>>>>>>>>>> detected during:<br>
> >>>>>>>>>>> instantiation of "void linearOut(void *, const T &)<br>
> >>>>>>>>>>> [with<br>
> >>>>>>>>>>> T=PetscInt={PetscInt64={long long}}]" at line 155 of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/EBToo<wbr>ls/MiniIVFABI.H"<br>
> >>>>>>>>>>> instantiation of "void MiniIVFAB<T>::linearOut(void<br>
> >>>>>>>>>>> *,<br>
> >>>>>>>>>>> const<br>
> >>>>>>>>>>> Box &, const Interval &) const [with T=PetscInt={PetscInt64={long<br>
> >>>>>>>>>>> long}}]"<br>
> >>>>>>>>>>> at line 169 of "../../../Chombo/lib/src/EBToo<br>
> >>>>>>>>>>> ls/BaseEBCellFABI.H"<br>
> >>>>>>>>>>> instantiation of "void BaseEBCellFAB<T>::linearOut(vo<br>
> >>>>>>>>>>> id<br>
> >>>>>>>>>>> *,<br>
> >>>>>>>>>>> const Box &, const Interval &) const [with<br>
> >>>>>>>>>>> T=PetscInt={PetscInt64={long<br>
> >>>>>>>>>>> long}}]" at line 154 of "../../../Chombo/lib/src/BoxTo<br>
> >>>>>>>>>>> ols/BoxLayoutData.H"<br>
> >>>>>>>>>>> instantiation of "void LDOperator<T>::linearOut(const<br>
> >>>>>>>>>>> T &,<br>
> >>>>>>>>>>> void *, const Box &, const Interval &) const [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>>>>>>>>> implicit generation of "LDOperator<T>::LDOperator()<br>
> >>>>>>>>>>> [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>>>>>>>>> instantiation of class "LDOperator<T> [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>>>>>>>>> instantiation of "void LevelData<T>::copyTo(const<br>
> >>>>>>>>>>> Interval &,<br>
> >>>>>>>>>>> BoxLayoutData<T> &, const Interval &, const Copier &, const<br>
> >>>>>>>>>>> LDOperator<T><br>
> >>>>>>>>>>> &) const [with T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long<br>
> >>>>>>>>>>> long}}>]"<br>
> >>>>>>>>>>> at<br>
> >>>>>>>>>>> line 213 of "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelDataI.H"<br>
> >>>>>>>>>>> instantiation of "void LevelData<T>::copyTo(const<br>
> >>>>>>>>>>> Interval &,<br>
> >>>>>>>>>>> BoxLayoutData<T> &, const Interval &) const [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 32<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelDataI.H"<br>
> >>>>>>>>>>> instantiation of "LevelData<T>::LevelData() [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 41<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../srcLib/EBPetscSolver.cpp"<br>
> >>>>>>>>>>><br>
> >>>>>>>>>>> ../../../Chombo/lib/src/BaseTo<wbr>ols/SPMDI.H(29): error: expression<br>
> >>>>>>>>>>> must<br>
> >>>>>>>>>>> have class type<br>
> >>>>>>>>>>> a_outputT.linearIn(inBuf);<br>
> >>>>>>>>>>> ^<br>
> >>>>>>>>>>> detected during:<br>
> >>>>>>>>>>> instantiation of "void linearIn(T &, const void *)<br>
> >>>>>>>>>>> [with<br>
> >>>>>>>>>>> T=PetscInt={PetscInt64={long long}}]" at line 181 of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/EBToo<wbr>ls/MiniIVFABI.H"<br>
> >>>>>>>>>>> instantiation of "void MiniIVFAB<T>::linearIn(void *,<br>
> >>>>>>>>>>> const<br>
> >>>>>>>>>>> Box &, const Interval &) [with T=PetscInt={PetscInt64={long<br>
> >>>>>>>>>>> long}}]"<br>
> >>>>>>>>>>> at<br>
> >>>>>>>>>>> line 181 of "../../../Chombo/lib/src/EBToo<wbr>ls/BaseEBCellFABI.H"<br>
> >>>>>>>>>>> instantiation of "void BaseEBCellFAB<T>::linearIn(voi<br>
> >>>>>>>>>>> d<br>
</div></div><span>> >>>>>>>>>>> *,<br>
> >>>>>>>>>>> const Box &, const Interval &) [with T=PetscInt={PetscInt64={long<br>
> >>>>>>>>>>> long}}]"<br>
</span>> >>>>>>>>>>> at line 159 of "../../../Chombo/lib/src/BoxTo<br>
> >>>>>>>>>>> ols/BoxLayoutData.H"<br>
<div><div class="m_-108618658402229371h5">> >>>>>>>>>>> instantiation of "void LDOperator<T>::linearIn(T &,<br>
> >>>>>>>>>>> void<br>
> >>>>>>>>>>> *,<br>
> >>>>>>>>>>> const Box &, const Interval &) const [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<br>
> >>>>>>>>>>> cInt64={long<br>
> >>>>>>>>>>> long}}>]" at line 114 of "../../../Chombo/lib/src/BoxTo<br>
> >>>>>>>>>>> ols/LevelData.H"<br>
> >>>>>>>>>>> implicit generation of "LDOperator<T>::LDOperator()<br>
> >>>>>>>>>>> [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>>>>>>>>> instantiation of class "LDOperator<T> [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 114<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelData.H"<br>
> >>>>>>>>>>> instantiation of "void LevelData<T>::copyTo(const<br>
> >>>>>>>>>>> Interval &,<br>
> >>>>>>>>>>> BoxLayoutData<T> &, const Interval &, const Copier &, const<br>
> >>>>>>>>>>> LDOperator<T><br>
> >>>>>>>>>>> &) const [with T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long<br>
> >>>>>>>>>>> long}}>]"<br>
> >>>>>>>>>>> at<br>
> >>>>>>>>>>> line 213 of "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelDataI.H"<br>
> >>>>>>>>>>> instantiation of "void LevelData<T>::copyTo(const<br>
> >>>>>>>>>>> Interval &,<br>
> >>>>>>>>>>> BoxLayoutData<T> &, const Interval &) const [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 32<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../../../Chombo/lib/src/BoxTo<wbr>ols/LevelDataI.H"<br>
> >>>>>>>>>>> instantiation of "LevelData<T>::LevelData() [with<br>
> >>>>>>>>>>> T=BaseEBCellFAB<PetscInt={Pets<wbr>cInt64={long long}}>]" at line 41<br>
> >>>>>>>>>>> of<br>
> >>>>>>>>>>> "../srcLib/EBPetscSolver.cpp"<br>
> >>>>>>>>>>><br>
> >>>>>>>>>>> compilation aborted for ../srcLib/EBPetscSolver.cpp (code 2)<br>
> >>>>>>>>>>> gmake[1]: *** [o/3d.Linux.64.CC.ftn.OPTHIGH.<br>
> >>>>>>>>>>> MPI.PETSC/EBPetscSolver.o]<br>
> >>>>>>>>>>> Error 2<br>
> >>>>>>>>>>> make: *** [all-example] Error 2<br>
> >>>>>>>>>>><br>
> >>>>>>>>>>><br>
> >>>>>>>>>>> On 12/19/2016 03:09 PM, Mark Adams wrote:<br>
> >>>>>>>>>>><br>
> >>>>>>>>>>> Sorry cleaning up. insert "petsc_install" in the path:<br>
> >>>>>>>>>>> "...madams/<br>
> >>>>>>>>>>><br>
> >>>>>>>>>>> petsc_install/petscv3.6..."<br>
> >>>>>>>>>>><br>
> >>>>>>>>>>>><br>
> >>>>>>>>>>>><br>
> >>>>>>>>>>>> On Mon, Dec 19, 2016 at 5:43 PM, David Trebotich <<br>
> >>>>>>>>>>>> <a href="mailto:treb@hpcrd.lbl.gov" target="_blank">treb@hpcrd.lbl.gov</a><br>
> >>>>>>>>>>>><br>
> >>>>>>>>>>>><br>
> >>>>>>>>>>>>> wrote:<br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>><br>
> >>>>>>>>>>>> Hey Mark-<br>
> >>>>>>>>>>>><br>
> >>>>>>>>>>>><br>
> >>>>>>>>>>>> I seem to have lost contact with petsc over the weekend:<br>
> >>>>>>>>>>>><br>
> >>>>>>>>>>>>> /global/homes/m/madams/petscv3<wbr>.6-xc30-opt64-intel//lib/<br>
> >>>>>>>>>>>>> petsc/conf/variables:<br>
> >>>>>>>>>>>>> No such file or directory<br>
> >>>>>>>>>>>>> Last chance .ex rule for target /global/homes/m/madams/petscv3<br>
> >>>>>>>>>>>>> .6-xc30-opt64-intel//lib/petsc<wbr>/conf/variables<br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>>> setenv | grep petsc:<br>
> >>>>>>>>>>>>> PE_PETSC_DEFAULT_VOLATILE_PKGC<wbr>ONFIG_PATH=/opt/cray/petsc/3.<br>
> >>>>>>>>>>>>> 7.2.0/real/@PRGENV@/@PE_PETSC_<wbr>DEFAULT_GENCOMPS@/@PE_PETSC_<br>
> >>>>>>>>>>>>> DEFAULT_TARGET@/lib/pkgconfig<br>
> >>>>>>>>>>>>> PETSC_DIR=/global/homes/m/mada<wbr>ms/petscv3.6-xc30-opt64-intel/<br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>>> Currently Loaded Modulefiles:<br>
> >>>>>>>>>>>>> 1) modules/<a href="http://3.2.10.4" rel="noreferrer" target="_blank">3.2.10.4</a><br>
</div></div>> >>>>>>>>>>>>> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__3.2.10.4&d=DQMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=Jd9TVPRE-Ts5OmoJe38KFIo29VVtsFIoN1ABdechuuE&s=UN73DjcLzn6LGQKK_kwoLCLWgyM5YEeKwwHYYCHe1nw&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint<wbr>.com/v2/url?u=http-3A__3.2.10.<wbr>4&d=DQMFaQ&c=54IZrppPQZKX9mLzc<wbr>GdPfFD1hxrcB__aEkJFOKJFd00&r=<wbr>DUUt3SRGI0_JgtNaS3udV68GRkgV4t<wbr>s7XKfj2opmiCY&m=Jd9TVPRE-Ts5Om<wbr>oJe38KFIo29VVtsFIoN1ABdechuuE&<wbr>s=UN73DjcLzn6LGQKK_kwoLCLWgyM5<wbr>YEeKwwHYYCHe1nw&e=</a>><br>
> >>>>>>>>>>>>> 8) intel/<a href="http://15.0.1.133" rel="noreferrer" target="_blank">15.0.1.133</a><br>
> >>>>>>>>>>>>> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__15.0.1.133&d=DQMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=Jd9TVPRE-Ts5OmoJe38KFIo29VVtsFIoN1ABdechuuE&s=Gix6rgPxH5UFQeKmDN6zAXMAmFezfJ1qY-723XuYrUE&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint<wbr>.com/v2/url?u=http-3A__15.0.1.<wbr>133&d=DQMFaQ&c=54IZrppPQZKX9mL<wbr>zcGdPfFD1hxrcB__aEkJFOKJFd00&<wbr>r=DUUt3SRGI0_JgtNaS3udV68GRkgV<wbr>4ts7XKfj2opmiCY&m=Jd9TVPRE-Ts5<wbr>OmoJe38KFIo29VVtsFIoN1ABdechuu<wbr>E&s=Gix6rgPxH5UFQeKmDN6zAXMAmF<wbr>ezfJ1qY-723XuYrUE&e=</a>><br>
> >>>>>>>>>>>>> 15) xpmem/<a href="http://0.1-2.0502.57015.1.15.ar" rel="noreferrer" target="_blank">0.1-2.0502.57015.1.15.ar</a><br>
> >>>>>>>>>>>>> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__0.1-2D2.0502.57015.1.15.ar&d=DQMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=Jd9TVPRE-Ts5OmoJe38KFIo29VVtsFIoN1ABdechuuE&s=DcDHbf2qyi7t-rquIoqAdDx5HPEa3BXjxxWGpgLVhBU&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint<wbr>.com/v2/url?u=http-3A__0.1-<wbr>2D2.0502.57015.1.15.ar&d=<wbr>DQMFaQ&c=54IZrppPQZKX9mLzcGdPf<wbr>FD1hxrcB__aEkJFOKJFd00&r=<wbr>DUUt3SRGI0_JgtNaS3udV68GRkgV4t<wbr>s7XKfj2opmiCY&m=Jd9TVPRE-Ts5Om<wbr>oJe38KFIo29VVtsFIoN1ABdechuuE&<wbr>s=DcDHbf2qyi7t-rquIoqAdDx5HPEa<wbr>3BXjxxWGpgLVhBU&e=</a>>i<br>
<span>> >>>>>>>>>>>>> 22)<br>
> >>>>>>>>>>>>> cray-mpich/7.4.1<br>
> >>>>>>>>>>>>> 2) nsg/1.2.0 9)<br>
> >>>>>>>>>>>>> cray-libsci/16.07.1<br>
> >>>>>>>>>>>>> 16) dvs/2.5_0.9.0-1.0502.1958.2.55<wbr>.ari 23)<br>
> >>>>>>>>>>>>> slurm/edison<br>
> >>>>>>>>>>>>> 3) eswrap/1.3.3-1.020200.1278.0 10)<br>
> >>>>>>>>>>>>> udreg/2.3.2-1.0502.9889.2.20.a<wbr>ri 17)<br>
> >>>>>>>>>>>>> alps/5.2.3-2.0502.9295.14.14.a<br>
> >>>>>>>>>>>>> ri<br>
> >>>>>>>>>>>>> 24) altd/2.0<br>
> >>>>>>>>>>>>> 4) switch/1.0-1.0502.57058.1.58.a<wbr>ri 11)<br>
> >>>>>>>>>>>>> ugni/6.0-1.0502.10245.9.9.ari 18) rca/<br>
> >>>>>>>>>>>>> <a href="http://1.0.0-2.0502.57212.2.56.ar" rel="noreferrer" target="_blank">1.0.0-2.0502.57212.2.56.ar</a><br>
</span>> >>>>>>>>>>>>> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__1.0.0-2D2.0502.57212.2.56.ar&d=DQMFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=Jd9TVPRE-Ts5OmoJe38KFIo29VVtsFIoN1ABdechuuE&s=KaOFCQYgi4MV721r1zVhLb666-2L2dmTfExdbRrrB1s&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint<wbr>.com/v2/url?u=http-3A__1.0.0-<wbr>2D2.0502.57212.2.56.ar&d=<wbr>DQMFaQ&c=54IZrppPQZKX9mLzcGdPf<wbr>FD1hxrcB__aEkJFOKJFd00&r=<wbr>DUUt3SRGI0_JgtNaS3udV68GRkgV4t<wbr>s7XKfj2opmiCY&m=Jd9TVPRE-Ts5Om<wbr>oJe38KFIo29VVtsFIoN1ABdechuuE&<wbr>s=KaOFCQYgi4MV721r1zVhLb666-2L<wbr>2dmTfExdbRrrB1s&e=</a>><br>
<div class="m_-108618658402229371HOEnZb"><div class="m_-108618658402229371h5">> >>>>>>>>>>>>> i<br>
> >>>>>>>>>>>>> 25) darshan/2.3.1<br>
> >>>>>>>>>>>>> 5) craype-ivybridge 12) pmi/<br>
> >>>>>>>>>>>>> 5.0.10-1.0000.11050.0.0.ari 19) atp/2.0.2 26)<br>
> >>>>>>>>>>>>> emacs/24.2<br>
> >>>>>>>>>>>>> 6) craype-network-aries 13)<br>
> >>>>>>>>>>>>> dmapp/7.0.1-1.0502.10246.8.47.<wbr>ari 20) PrgEnv-intel/5.2.56<br>
> >>>>>>>>>>>>> 27)<br>
> >>>>>>>>>>>>> cray-hdf5-parallel/1.8.16<br>
> >>>>>>>>>>>>> 7) craype/2.4.2 14)<br>
> >>>>>>>>>>>>> gni-headers/4.0-1.0502.10317.9<wbr>.2.ari 21) cray-shmem/7.4.1<br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>>> Thanks.<br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>>> treb<br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>>> --<br>
> >>>>>>>>>>>>> David Trebotich<br>
> >>>>>>>>>>>>> Applied Numerical Algorithms Group<br>
> >>>>>>>>>>>>> Lawrence Berkeley National Laboratory<br>
> >>>>>>>>>>>>> <a href="tel:(510)%20486-5984" value="+15104865984" target="_blank">(510) 486-5984</a> <%28510%29%20486-5984> office<br>
> >>>>>>>>>>>>> <a href="tel:(510)%20384-6868" value="+15103846868" target="_blank">(510) 384-6868</a> <%28510%29%20384-6868> mobile<br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>>> --<br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>>><br>
> >>>>>>>>>>>> David Trebotich<br>
> >>>>>>>>>>>><br>
> >>>>>>>>>>> Applied Numerical Algorithms Group<br>
> >>>>>>>>>>> Lawrence Berkeley National Laboratory<br>
> >>>>>>>>>>> <a href="tel:(510)%20486-5984" value="+15104865984" target="_blank">(510) 486-5984</a> <%28510%29%20486-5984> office<br>
> >>>>>>>>>>> <a href="tel:(510)%20384-6868" value="+15103846868" target="_blank">(510) 384-6868</a> <%28510%29%20384-6868> mobile<br>
> >>>>>>>>>>><br>
> >>>>>>>>>>><br>
> >>>>>>>>>>><br>
> >>>>>>>>>>><br>
> >>>>>>>>>>> --<br>
> >>>>>>>>>>><br>
> >>>>>>>>>><br>
> >>>>>>>>>> David Trebotich<br>
> >>>>>>>>> Applied Numerical Algorithms Group<br>
> >>>>>>>>> Lawrence Berkeley National Laboratory<br>
> >>>>>>>>> <a href="tel:(510)%20486-5984" value="+15104865984" target="_blank">(510) 486-5984</a> <%28510%29%20486-5984> office<br>
> >>>>>>>>> <a href="tel:(510)%20384-6868" value="+15103846868" target="_blank">(510) 384-6868</a> <%28510%29%20384-6868> mobile<br>
> >>>>>>>>><br>
> >>>>>>>>><br>
> >>>>>>>>><br>
> >>>>>>>>><br>
> >>>>>>>>> --<br>
> >>>>>>>><br>
> >>>>>>> David Trebotich<br>
> >>>>>>> Applied Numerical Algorithms Group<br>
> >>>>>>> Lawrence Berkeley National Laboratory<br>
> >>>>>>> <a href="tel:(510)%20486-5984" value="+15104865984" target="_blank">(510) 486-5984</a> <%28510%29%20486-5984> office<br>
> >>>>>>> <a href="tel:(510)%20384-6868" value="+15103846868" target="_blank">(510) 384-6868</a> <%28510%29%20384-6868> mobile<br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>><br>
> >>>>> --<br>
> >>>> David Trebotich<br>
> >>>> Applied Numerical Algorithms Group<br>
> >>>> Lawrence Berkeley National Laboratory<br>
> >>>> <a href="tel:(510)%20486-5984" value="+15104865984" target="_blank">(510) 486-5984</a> office<br>
> >>>> <a href="tel:(510)%20384-6868" value="+15103846868" target="_blank">(510) 384-6868</a> mobile<br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>> --<br>
> >>> David Trebotich<br>
> >>> Applied Numerical Algorithms Group<br>
> >>> Computational Research Division<br>
> >>> Lawrence Berkeley National Laboratory<a href="tel:(510)%20486-5984" value="+15104865984" target="_blank">(510) 486-5984</a> office<a href="tel:(510)%20384-6868" value="+15103846868" target="_blank">(510) 384-6868</a><br>
> >>> mobile<br>
> >>><br>
> >>><br>
> >>><br>
> >><br>
> > --<br>
> > David Trebotich<br>
> > Applied Numerical Algorithms Group<br>
> > Lawrence Berkeley National Laboratory<br>
> > <a href="tel:(510)%20486-5984" value="+15104865984" target="_blank">(510) 486-5984</a> office<br>
> > <a href="tel:(510)%20384-6868" value="+15103846868" target="_blank">(510) 384-6868</a> mobile<br>
> ><br>
> ><br>
> ><br>
><br>
<br>
</div></div></blockquote></div><br></div></div>