[ExM Users] Swift/T using wrong Python
Tim Armstrong
tim.g.armstrong at gmail.com
Tue Apr 21 16:57:18 CDT 2015
Hmm, I got it working with the following settings:
ENABLE_PYTHON=1
PYTHON_INSTALL=/home/tim/anaconda
I rebuilt everything from scratch with the rebuild-all.sh script, then was
able to get it to report the correct version:
$ PYTHONHOME=$HOME/anaconda TURBINE_LOG=0 swift-t py-version.swift
2.7.9 |Anaconda 2.2.0 (64-bit)| (default, Mar 9 2015, 16:20:48)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
py-version.swift is just:
import io;
import python;
printf(python("import sys\nsys.version"));
On 21 April 2015 at 11:23, Matthew Turk <matthewturk at gmail.com> wrote:
> Hi all,
>
> I set up Swift/T for the first time the other day and had similar
> problems getting it to use my custom-built Python. I was able to get
> exm-settings.sh to be recognized, but then during execution it failed.
> I ended up having to set both PYTHONHOME *and* LD_PRELOAD, where
> LD_PRELOAD had to have both the MPI .so file and the libpython2.7.so
> included. After that it worked just fine.
>
> -Matt
>
> On Tue, Apr 21, 2015 at 11:19 AM, Ozik, Jonathan <jozik at anl.gov> wrote:
> > Tim,
> >
> > I’ve tried to set the PYTHONHOME variable a few times to see if I can
> force
> > the right python version to be found but I have yet to be successful.
> >
> > Jonathan
> >
> > On Apr 21, 2015, at 9:54 AM, Tim Armstrong <tim.g.armstrong at gmail.com>
> > wrote:
> >
> > It looks like the environment variable PYTHONHOME=/path/to/anaconda
> needs to
> > be set for it to look in the right places for the packages. I'm looking
> to
> > see if we can automatically set this up so that it does the right thing
> in
> > future.
> >
> > - Tim
> >
> > On 20 April 2015 at 18:24, Ozik, Jonathan <jozik at anl.gov> wrote:
> >>
> >> Tim,
> >>
> >> Just for kicks I tried to create a standalone conda environment for
> Python
> >> 2.7 and used that as my PYTHON_INSTALL location but I still am seeing
> >> incorrect sys.version, platform.python_version(), etc. Not sure what’s
> going
> >> on, but like I said earlier, it doesn’t seem to affect the running of
> the
> >> code.
> >>
> >> Jonathan
> >>
> >> On Apr 20, 2015, at 4:56 PM, Jonathan Ozik <jozik at anl.gov> wrote:
> >>
> >> Tim,
> >>
> >> I had:
> >> PYTHON_INSTALL=/Users/jozik/anaconda
> >>
> >> I had to go in and undo some of the attempts I had made to fix the issue
> >> (but probably just exacerbated it), and now with everything back to its
> >> initial settings I get the intended:
> >> Using Python lib directory: /Users/jozik/anaconda/lib
> >> Using Python lib name: python2.7
> >>
> >> If I ignore what I’m seeing returned by sys.version, it does look like
> the
> >> correct Python version is being picked up, at least based on the modules
> >> that are available. Maybe the sys.version issue is associated with the
> >> subtle issue you mentioned?
> >>
> >> In any case, thank you for the help.
> >>
> >> Jonathan
> >>
> >> On Apr 20, 2015, at 9:46 AM, Tim Armstrong <tim.g.armstrong at gmail.com>
> >> wrote:
> >>
> >> What did you have PYTHON_INSTALL set to in exm-settings? It should be
> >> printing the directory/name there.
> >>
> >> I was actually able to replicate a similar problem, which may not be
> what
> >> you were seeing. There's a subtle issue with ordering of library
> >> dependencies where it was picking up the system Python library rather
> than
> >> the other one in some cases.
> >>
> >> - Tim
> >>
> >> On 19 April 2015 at 23:28, Ozik, Jonathan <jozik at anl.gov> wrote:
> >>>
> >>> I’m not an Autoconf expert, so take this with a grain of salt, but the
> >>> two lines in the turbine configure script:
> >>> echo "Using Python lib directory: ${USE_PYTHON_LIBDIR}"
> >>> echo "Using Python lib name: ${USE_PYTHON_NAME}”
> >>>
> >>> were showing:
> >>> Using Python lib directory: .
> >>> Using Python lib name: 1
> >>>
> >>> so I looked at how those variables were being derived, and it looks
> like
> >>> there might be some issues there.
> >>> Also, exm-settings.sh doesn’t mention the --with-python-lib option
> >>> either.
> >>>
> >>> As for the sys.version issue that I’m seeing, it’s really strange,
> >>> because it is possible that sys.version is not correct, as odd as that
> may
> >>> sound. I know that sys.executable is not always correct, but I thought
> I
> >>> could trust sys.version. The python that is being used definitely has
> >>> modules installed that are not in the version of python that’s being
> >>> reported. Odd. But the good news is that I’m now able to move on with
> what
> >>> we’re actually trying to do.
> >>>
> >>> Thanks again for helping me think through this,
> >>>
> >>> Jonathan
> >>>
> >>> On Apr 19, 2015, at 9:21 PM, Tim Armstrong <tim.g.armstrong at gmail.com>
> >>> wrote:
> >>>
> >>> Hmm, so it's probably picking up the system Python by default. This
> may
> >>> be an oversight on our part - I'm creating a bug to look into it. Our
> build
> >>> should be linking it to your other python library but isn't.
> >>>
> >>> As a workaround, you can override this with the DYLD_LIBRARY_PATH
> >>> environment variable on Mac OS X.
> >>>
> >>> If you add the directory with your preferred python .dylib file to that
> >>> environment variable, it should link against that version of python
> instead
> >>> of the system one.
> >>>
> >>> E.g.
> >>>
> >>> export DYLD_LIBRARY_PATH=/path/to/anaconda/lib
> >>>
> >>> I *think* this should give you a way to work around it in the meantime.
> >>>
> >>> - Tim
> >>>
> >>> On 19 April 2015 at 20:19, Ozik, Jonathan <jozik at anl.gov> wrote:
> >>>>
> >>>> Thanks Tim. Here’s what I get on OS X:
> >>>>
> >>>> XioMBP:swift_simphony_example_DEAP jozik$ otool -L
> >>>> /Users/jozik/work/swift_t/install/turbine/lib/libtclturbine.dylib
> >>>> /Users/jozik/work/swift_t/install/turbine/lib/libtclturbine.dylib:
> >>>> /Users/jozik/work/swift_t/install/turbine/lib/libtclturbine.dylib
> >>>> (compatibility version 0.0.0, current version 0.0.0)
> >>>> /Users/jozik/work/swift_t/install/lb/lib/libadlb.dylib (compatibility
> >>>> version 0.0.0, current version 0.0.0)
> >>>> /Users/jozik/work/swift_t/install/c-utils/lib/libexmcutils.dylib
> >>>> (compatibility version 0.0.0, current version 0.0.0)
> >>>> /usr/local/opt/tcl-tk/lib/libtcl8.6.dylib (compatibility version
> 8.6.0,
> >>>> current version 8.6.3)
> >>>> /usr/local/lib/libmpi.12.dylib (compatibility version 13.0.0, current
> >>>> version 13.4.0)
> >>>> libpython2.7.dylib (compatibility version 2.7.0, current version
> 2.7.0)
> >>>> libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
> >>>> /usr/local/lib/libsz.2.dylib (compatibility version 3.0.0, current
> >>>> version 3.0.0)
> >>>> /usr/local/lib/libhdf5.8.dylib (compatibility version 9.0.0, current
> >>>> version 9.2.0)
> >>>> /usr/local/lib/libhdf5_hl.8.dylib (compatibility version 9.0.0,
> current
> >>>> version 9.2.0)
> >>>> /usr/local/lib/libpmpi.12.dylib (compatibility version 13.0.0, current
> >>>> version 13.4.0)
> >>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version
> >>>> 1213.0.0)
> >>>>
> >>>>
> /usr/local/Cellar/gcc/4.9.2/lib/gcc/x86_64-apple-darwin14.0.0/4.9.2/libgcc_s.1.dylib
> >>>> (compatibility version 1.0.0, current version 1.0.0)
> >>>>
> >>>> Which doesn’t answer our question of which libpython2.7.dylib is being
> >>>> linked to… If there’s a build log file that you want to point me to,
> I can
> >>>> do some more digging.
> >>>>
> >>>> Jonathan
> >>>>
> >>>> On Apr 19, 2015, at 7:36 PM, Tim Armstrong <tim.g.armstrong at gmail.com
> >
> >>>> wrote:
> >>>>
> >>>> It might be helpful to get some linker information from your
> >>>> installation to see which Python version it's linking against.
> >>>>
> >>>> In linux:
> >>>> ldd /path/to/turbine/lib/libtclturbine.so
> >>>>
> >>>> In OS X I think it's something like:
> >>>> otool -L /path/to/turbine/lib/libtclturbine.dylib
> >>>>
> >>>> - Tim
> >>>>
> >>>> On 19 April 2015 at 15:55, Ozik, Jonathan <jozik at anl.gov> wrote:
> >>>>>
> >>>>> An update. I looked to see whether this was a PATH issue, and saw
> that
> >>>>> there was a difference between how my bash and zsh shells populate
> the PATH
> >>>>> variable. I have made them compatible now, but am still seeing the
> same
> >>>>> issue with the wrong python being picked up.
> >>>>>
> >>>>> Jonathan
> >>>>>
> >>>>> > On Apr 18, 2015, at 11:21 PM, Ozik, Jonathan <jozik at anl.gov>
> wrote:
> >>>>> >
> >>>>> > Hi all,
> >>>>> >
> >>>>> > I’ve run up against an issue where Swift/T’s python and
> >>>>> > python_persist are using the wrong Python version. I’ve checked
> and rebuilt
> >>>>> > and it looks like the Python install location specified in
> exm-settings.sh
> >>>>> > is being used in the build log but when I run a simple test:
> >>>>> > string ss = python_persist("import sys\nsys.version”);
> >>>>> > I’m getting the Python installation at /usr/bin/python rather than
> >>>>> > the Python version installed as part of the Anaconda distribution
> (which is
> >>>>> > what the exm-settings.sh is configured to use).
> >>>>> > Any advice on how I might better diagnose this would be
> appreciated.
> >>>>> > I’m on Mac OS X (10.10.3).
> >>>>> >
> >>>>> > Jonathan
> >>>>> >
> >>>>> > _______________________________________________
> >>>>> > ExM-user mailing list
> >>>>> > ExM-user at lists.mcs.anl.gov
> >>>>> > https://lists.mcs.anl.gov/mailman/listinfo/exm-user
> >>>>>
> >>>>> _______________________________________________
> >>>>> ExM-user mailing list
> >>>>> ExM-user at lists.mcs.anl.gov
> >>>>> https://lists.mcs.anl.gov/mailman/listinfo/exm-user
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >
> > _______________________________________________
> > ExM-user mailing list
> > ExM-user at lists.mcs.anl.gov
> > https://lists.mcs.anl.gov/mailman/listinfo/exm-user
> >
> >
> >
> > _______________________________________________
> > ExM-user mailing list
> > ExM-user at lists.mcs.anl.gov
> > https://lists.mcs.anl.gov/mailman/listinfo/exm-user
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/exm-user/attachments/20150421/5d15eac5/attachment.html>
More information about the ExM-user
mailing list