[ExM Users] Swift/T using wrong Python

Tim Armstrong tim.g.armstrong at gmail.com
Tue Apr 21 17:08:16 CDT 2015 

Yep, I have a python2.7 install in /usr/bin .  It was picking that up if I
ran without setting PYTHONHOME

Justin I think has easier access to a Mac than I do.

- Tim

On 21 April 2015 at 17:01, Ozik, Jonathan <jozik at anl.gov> wrote:

>  So there’s another python executable in /usr/bin?
> Do you have access to a machine with Mac OS X? It could be a platform
> idiosyncrasy.
>
>  On Apr 21, 2015, at 4:57 PM, Tim Armstrong <tim.g.armstrong at gmail.com>
> wrote:
>
>  Hmm, I got it working with the following settings:
>
> ENABLE_PYTHON=1
> PYTHON_INSTALL=/home/tim/anaconda
>
>  I rebuilt everything from scratch with the rebuild-all.sh script, then
> was able to get it to report the correct version:
>
> $ PYTHONHOME=$HOME/anaconda TURBINE_LOG=0 swift-t py-version.swift
>
> 2.7.9 |Anaconda 2.2.0 (64-bit)| (default, Mar  9 2015, 16:20:48)
> [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
>
>  py-version.swift is just:
>
> import io;
> import python;
>
> printf(python("import sys\nsys.version"));
>
>
>
>
> On 21 April 2015 at 11:23, Matthew Turk <matthewturk at gmail.com> wrote:
>
>> Hi all,
>>
>> I set up Swift/T for the first time the other day and had similar
>> problems getting it to use my custom-built Python.  I was able to get
>> exm-settings.sh to be recognized, but then during execution it failed.
>> I ended up having to set both PYTHONHOME *and* LD_PRELOAD, where
>> LD_PRELOAD had to have both the MPI .so file and the libpython2.7.so
>> included.  After that it worked just fine.
>>
>> -Matt
>>
>> On Tue, Apr 21, 2015 at 11:19 AM, Ozik, Jonathan <jozik at anl.gov> wrote:
>> > Tim,
>> >
>> > I’ve tried to set the PYTHONHOME variable a few times to see if I can
>> force
>> > the right python version to be found but I have yet to be successful.
>> >
>> > Jonathan
>> >
>> > On Apr 21, 2015, at 9:54 AM, Tim Armstrong <tim.g.armstrong at gmail.com>
>> > wrote:
>> >
>> > It looks like the environment variable PYTHONHOME=/path/to/anaconda
>> needs to
>> > be set for it to look in the right places for the packages.  I'm
>> looking to
>> > see if we can automatically set this up so that it does the right thing
>> in
>> > future.
>> >
>> > - Tim
>> >
>> > On 20 April 2015 at 18:24, Ozik, Jonathan <jozik at anl.gov> wrote:
>> >>
>> >> Tim,
>> >>
>> >> Just for kicks I tried to create a standalone conda environment for
>> Python
>> >> 2.7 and used that as my PYTHON_INSTALL location but I still am seeing
>> >> incorrect sys.version, platform.python_version(), etc. Not sure what’s
>> going
>> >> on, but like I said earlier, it doesn’t seem to affect the running of
>> the
>> >> code.
>> >>
>> >> Jonathan
>> >>
>> >> On Apr 20, 2015, at 4:56 PM, Jonathan Ozik <jozik at anl.gov> wrote:
>> >>
>> >> Tim,
>> >>
>> >> I had:
>> >> PYTHON_INSTALL=/Users/jozik/anaconda
>> >>
>> >> I had to go in and undo some of the attempts I had made to fix the
>> issue
>> >> (but probably just exacerbated it), and now with everything back to its
>> >> initial settings I get the intended:
>> >> Using Python lib directory: /Users/jozik/anaconda/lib
>> >> Using Python lib name:      python2.7
>> >>
>> >> If I ignore what I’m seeing returned by sys.version, it does look like
>> the
>> >> correct Python version is being picked up, at least based on the
>> modules
>> >> that are available. Maybe the sys.version issue is associated with the
>> >> subtle issue you mentioned?
>> >>
>> >> In any case, thank you for the help.
>> >>
>> >> Jonathan
>> >>
>> >> On Apr 20, 2015, at 9:46 AM, Tim Armstrong <tim.g.armstrong at gmail.com>
>> >> wrote:
>> >>
>> >> What did you have PYTHON_INSTALL set to in exm-settings?  It should be
>> >> printing the directory/name there.
>> >>
>> >> I was actually able to replicate a similar problem, which may not be
>> what
>> >> you were seeing.  There's a subtle issue with ordering of library
>> >> dependencies where it was picking up the system Python library rather
>> than
>> >> the other one in some cases.
>> >>
>> >> - Tim
>> >>
>> >> On 19 April 2015 at 23:28, Ozik, Jonathan <jozik at anl.gov> wrote:
>> >>>
>> >>> I’m not an Autoconf expert, so take this with a grain of salt, but the
>> >>> two lines in the turbine configure script:
>> >>>     echo "Using Python lib directory: ${USE_PYTHON_LIBDIR}"
>> >>>     echo "Using Python lib name:      ${USE_PYTHON_NAME}”
>> >>>
>> >>> were showing:
>> >>> Using Python lib directory: .
>> >>> Using Python lib name:      1
>> >>>
>> >>> so I looked at how those variables were being derived, and it looks
>> like
>> >>> there might be some issues there.
>> >>> Also, exm-settings.sh doesn’t mention the --with-python-lib option
>> >>> either.
>> >>>
>> >>> As for the sys.version issue that I’m seeing, it’s really strange,
>> >>> because it is possible that sys.version is not correct, as odd as
>> that may
>> >>> sound. I know that sys.executable is not always correct, but I
>> thought I
>> >>> could trust sys.version. The python that is being used definitely has
>> >>> modules installed that are not in the version of python that’s being
>> >>> reported. Odd. But the good news is that I’m now able to move on with
>> what
>> >>> we’re actually trying to do.
>> >>>
>> >>> Thanks again for helping me think through this,
>> >>>
>> >>> Jonathan
>> >>>
>> >>> On Apr 19, 2015, at 9:21 PM, Tim Armstrong <tim.g.armstrong at gmail.com
>> >
>> >>> wrote:
>> >>>
>> >>> Hmm, so it's probably picking up the system Python by default.   This
>> may
>> >>> be an oversight on our part - I'm creating a bug to look into it.
>> Our build
>> >>> should be linking it to your other python library but isn't.
>> >>>
>> >>> As a workaround, you can override this with the DYLD_LIBRARY_PATH
>> >>> environment variable on Mac OS X.
>> >>>
>> >>> If you add the directory with your preferred python .dylib file to
>> that
>> >>> environment variable, it should link against that version of python
>> instead
>> >>> of the system one.
>> >>>
>> >>> E.g.
>> >>>
>> >>> export DYLD_LIBRARY_PATH=/path/to/anaconda/lib
>> >>>
>> >>> I *think* this should give you a way to work around it in the
>> meantime.
>> >>>
>> >>> - Tim
>> >>>
>> >>> On 19 April 2015 at 20:19, Ozik, Jonathan <jozik at anl.gov> wrote:
>> >>>>
>> >>>> Thanks Tim. Here’s what I get on OS X:
>> >>>>
>> >>>> XioMBP:swift_simphony_example_DEAP jozik$ otool -L
>> >>>> /Users/jozik/work/swift_t/install/turbine/lib/libtclturbine.dylib
>> >>>> /Users/jozik/work/swift_t/install/turbine/lib/libtclturbine.dylib:
>> >>>> /Users/jozik/work/swift_t/install/turbine/lib/libtclturbine.dylib
>> >>>> (compatibility version 0.0.0, current version 0.0.0)
>> >>>> /Users/jozik/work/swift_t/install/lb/lib/libadlb.dylib (compatibility
>> >>>> version 0.0.0, current version 0.0.0)
>> >>>> /Users/jozik/work/swift_t/install/c-utils/lib/libexmcutils.dylib
>> >>>> (compatibility version 0.0.0, current version 0.0.0)
>> >>>> /usr/local/opt/tcl-tk/lib/libtcl8.6.dylib (compatibility version
>> 8.6.0,
>> >>>> current version 8.6.3)
>> >>>> /usr/local/lib/libmpi.12.dylib (compatibility version 13.0.0, current
>> >>>> version 13.4.0)
>> >>>> libpython2.7.dylib (compatibility version 2.7.0, current version
>> 2.7.0)
>> >>>> libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
>> >>>> /usr/local/lib/libsz.2.dylib (compatibility version 3.0.0, current
>> >>>> version 3.0.0)
>> >>>> /usr/local/lib/libhdf5.8.dylib (compatibility version 9.0.0, current
>> >>>> version 9.2.0)
>> >>>> /usr/local/lib/libhdf5_hl.8.dylib (compatibility version 9.0.0,
>> current
>> >>>> version 9.2.0)
>> >>>> /usr/local/lib/libpmpi.12.dylib (compatibility version 13.0.0,
>> current
>> >>>> version 13.4.0)
>> >>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
>> version
>> >>>> 1213.0.0)
>> >>>>
>> >>>>
>> /usr/local/Cellar/gcc/4.9.2/lib/gcc/x86_64-apple-darwin14.0.0/4.9.2/libgcc_s.1.dylib
>> >>>> (compatibility version 1.0.0, current version 1.0.0)
>> >>>>
>> >>>> Which doesn’t answer our question of which libpython2.7.dylib is
>> being
>> >>>> linked to… If there’s a build log file that you want to point me to,
>> I can
>> >>>> do some more digging.
>> >>>>
>> >>>> Jonathan
>> >>>>
>> >>>> On Apr 19, 2015, at 7:36 PM, Tim Armstrong <
>> tim.g.armstrong at gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> It might be helpful to get some linker information from your
>> >>>> installation to see which Python version it's linking against.
>> >>>>
>> >>>> In linux:
>> >>>> ldd  /path/to/turbine/lib/libtclturbine.so
>> >>>>
>> >>>> In OS X I think it's something like:
>> >>>> otool -L /path/to/turbine/lib/libtclturbine.dylib
>> >>>>
>> >>>> - Tim
>> >>>>
>> >>>> On 19 April 2015 at 15:55, Ozik, Jonathan <jozik at anl.gov> wrote:
>> >>>>>
>> >>>>> An update. I looked to see whether this was a PATH issue, and saw
>> that
>> >>>>> there was a difference between how my bash and zsh shells populate
>> the PATH
>> >>>>> variable. I have made them compatible now, but am still seeing the
>> same
>> >>>>> issue with the wrong python being picked up.
>> >>>>>
>> >>>>> Jonathan
>> >>>>>
>> >>>>> > On Apr 18, 2015, at 11:21 PM, Ozik, Jonathan <jozik at anl.gov>
>> wrote:
>> >>>>> >
>> >>>>> > Hi all,
>> >>>>> >
>> >>>>> > I’ve run up against an issue where Swift/T’s python and
>> >>>>> > python_persist are using the wrong Python version. I’ve checked
>> and rebuilt
>> >>>>> > and it looks like the Python install location specified in
>> exm-settings.sh
>> >>>>> > is being used in the build log but when I run a simple test:
>> >>>>> > string ss = python_persist("import sys\nsys.version”);
>> >>>>> > I’m getting the Python installation at /usr/bin/python rather than
>> >>>>> > the Python version installed as part of the Anaconda distribution
>> (which is
>> >>>>> > what the exm-settings.sh is configured to use).
>> >>>>> > Any advice on how I might better diagnose this would be
>> appreciated.
>> >>>>> > I’m on Mac OS X (10.10.3).
>> >>>>> >
>> >>>>> > Jonathan
>> >>>>> >
>> >>>>> > _______________________________________________
>> >>>>> > ExM-user mailing list
>> >>>>> > ExM-user at lists.mcs.anl.gov
>> >>>>> > https://lists.mcs.anl.gov/mailman/listinfo/exm-user
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> ExM-user mailing list
>> >>>>> ExM-user at lists.mcs.anl.gov
>> >>>>> https://lists.mcs.anl.gov/mailman/listinfo/exm-user
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >
>> > _______________________________________________
>> > ExM-user mailing list
>> > ExM-user at lists.mcs.anl.gov
>> > https://lists.mcs.anl.gov/mailman/listinfo/exm-user
>> >
>> >
>> >
>> > _______________________________________________
>> > ExM-user mailing list
>> > ExM-user at lists.mcs.anl.gov
>> > https://lists.mcs.anl.gov/mailman/listinfo/exm-user
>> >
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/exm-user/attachments/20150421/7f97fe06/attachment-0001.html>

More information about the ExM-user mailing list