[ExM Users] Swift/T using wrong Python

Tue Apr 21 11:19:34 CDT 2015

Tim,

I’ve tried to set the PYTHONHOME variable a few times to see if I can force the right python version to be found but I have yet to be successful.

Jonathan

On Apr 21, 2015, at 9:54 AM, Tim Armstrong <tim.g.armstrong at gmail.com<mailto:tim.g.armstrong at gmail.com>> wrote:

It looks like the environment variable PYTHONHOME=/path/to/anaconda needs to be set for it to look in the right places for the packages.  I'm looking to see if we can automatically set this up so that it does the right thing in future.

- Tim

On 20 April 2015 at 18:24, Ozik, Jonathan <jozik at anl.gov<mailto:jozik at anl.gov>> wrote:
Tim,

Just for kicks I tried to create a standalone conda environment for Python 2.7 and used that as my PYTHON_INSTALL location but I still am seeing incorrect sys.version, platform.python_version(), etc. Not sure what’s going on, but like I said earlier, it doesn’t seem to affect the running of the code.

Jonathan

On Apr 20, 2015, at 4:56 PM, Jonathan Ozik <jozik at anl.gov<mailto:jozik at anl.gov>> wrote:

Tim,

I had:
PYTHON_INSTALL=/Users/jozik/anaconda

I had to go in and undo some of the attempts I had made to fix the issue (but probably just exacerbated it), and now with everything back to its initial settings I get the intended:
Using Python lib directory: /Users/jozik/anaconda/lib
Using Python lib name:      python2.7

If I ignore what I’m seeing returned by sys.version, it does look like the correct Python version is being picked up, at least based on the modules that are available. Maybe the sys.version issue is associated with the subtle issue you mentioned?

In any case, thank you for the help.

Jonathan

On Apr 20, 2015, at 9:46 AM, Tim Armstrong <tim.g.armstrong at gmail.com<mailto:tim.g.armstrong at gmail.com>> wrote:

What did you have PYTHON_INSTALL set to in exm-settings?  It should be printing the directory/name there.

I was actually able to replicate a similar problem, which may not be what you were seeing.  There's a subtle issue with ordering of library dependencies where it was picking up the system Python library rather than the other one in some cases.

- Tim

On 19 April 2015 at 23:28, Ozik, Jonathan <jozik at anl.gov<mailto:jozik at anl.gov>> wrote:
I’m not an Autoconf expert, so take this with a grain of salt, but the two lines in the turbine configure script:
    echo "Using Python lib directory: ${USE_PYTHON_LIBDIR}"
    echo "Using Python lib name:      ${USE_PYTHON_NAME}”

were showing:
Using Python lib directory: .
Using Python lib name:      1

so I looked at how those variables were being derived, and it looks like there might be some issues there.
Also, exm-settings.sh doesn’t mention the --with-python-lib option either.

As for the sys.version issue that I’m seeing, it’s really strange, because it is possible that sys.version is not correct, as odd as that may sound. I know that sys.executable is not always correct, but I thought I could trust sys.version. The python that is being used definitely has modules installed that are not in the version of python that’s being reported. Odd. But the good news is that I’m now able to move on with what we’re actually trying to do.

Thanks again for helping me think through this,

Jonathan

On Apr 19, 2015, at 9:21 PM, Tim Armstrong <tim.g.armstrong at gmail.com<mailto:tim.g.armstrong at gmail.com>> wrote:

Hmm, so it's probably picking up the system Python by default.   This may be an oversight on our part - I'm creating a bug to look into it.  Our build should be linking it to your other python library but isn't.

As a workaround, you can override this with the DYLD_LIBRARY_PATH environment variable on Mac OS X.

If you add the directory with your preferred python .dylib file to that environment variable, it should link against that version of python instead of the system one.

E.g.

export DYLD_LIBRARY_PATH=/path/to/anaconda/lib

I *think* this should give you a way to work around it in the meantime.

- Tim

On 19 April 2015 at 20:19, Ozik, Jonathan <jozik at anl.gov<mailto:jozik at anl.gov>> wrote:
Thanks Tim. Here’s what I get on OS X:

XioMBP:swift_simphony_example_DEAP jozik$ otool -L /Users/jozik/work/swift_t/install/turbine/lib/libtclturbine.dylib
/Users/jozik/work/swift_t/install/turbine/lib/libtclturbine.dylib:
/Users/jozik/work/swift_t/install/turbine/lib/libtclturbine.dylib (compatibility version 0.0.0, current version 0.0.0)
/Users/jozik/work/swift_t/install/lb/lib/libadlb.dylib (compatibility version 0.0.0, current version 0.0.0)
/Users/jozik/work/swift_t/install/c-utils/lib/libexmcutils.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/opt/tcl-tk/lib/libtcl8.6.dylib (compatibility version 8.6.0, current version 8.6.3)
/usr/local/lib/libmpi.12.dylib (compatibility version 13.0.0, current version 13.4.0)
libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
/usr/local/lib/libsz.2.dylib (compatibility version 3.0.0, current version 3.0.0)
/usr/local/lib/libhdf5.8.dylib (compatibility version 9.0.0, current version 9.2.0)
/usr/local/lib/libhdf5_hl.8.dylib (compatibility version 9.0.0, current version 9.2.0)
/usr/local/lib/libpmpi.12.dylib (compatibility version 13.0.0, current version 13.4.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/usr/local/Cellar/gcc/4.9.2/lib/gcc/x86_64-apple-darwin14.0.0/4.9.2/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

Which doesn’t answer our question of which libpython2.7.dylib is being linked to… If there’s a build log file that you want to point me to, I can do some more digging.

Jonathan

On Apr 19, 2015, at 7:36 PM, Tim Armstrong <tim.g.armstrong at gmail.com<mailto:tim.g.armstrong at gmail.com>> wrote:

It might be helpful to get some linker information from your installation to see which Python version it's linking against.

In linux:
ldd  /path/to/turbine/lib/libtclturbine.so

In OS X I think it's something like:
otool -L /path/to/turbine/lib/libtclturbine.dylib

- Tim

On 19 April 2015 at 15:55, Ozik, Jonathan <jozik at anl.gov<mailto:jozik at anl.gov>> wrote:
An update. I looked to see whether this was a PATH issue, and saw that there was a difference between how my bash and zsh shells populate the PATH variable. I have made them compatible now, but am still seeing the same issue with the wrong python being picked up.

Jonathan

> On Apr 18, 2015, at 11:21 PM, Ozik, Jonathan <jozik at anl.gov<mailto:jozik at anl.gov>> wrote:
>
> Hi all,
>
> I’ve run up against an issue where Swift/T’s python and python_persist are using the wrong Python version. I’ve checked and rebuilt and it looks like the Python install location specified in exm-settings.sh is being used in the build log but when I run a simple test:
> string ss = python_persist("import sys\nsys.version”);
> I’m getting the Python installation at /usr/bin/python rather than the Python version installed as part of the Anaconda distribution (which is what the exm-settings.sh is configured to use).
> Any advice on how I might better diagnose this would be appreciated. I’m on Mac OS X (10.10.3).
>
> Jonathan
>
> _______________________________________________
> ExM-user mailing list
> ExM-user at lists.mcs.anl.gov<mailto:ExM-user at lists.mcs.anl.gov>
> https://lists.mcs.anl.gov/mailman/listinfo/exm-user

_______________________________________________
ExM-user mailing list
ExM-user at lists.mcs.anl.gov<mailto:ExM-user at lists.mcs.anl.gov>
https://lists.mcs.anl.gov/mailman/listinfo/exm-user

_______________________________________________
ExM-user mailing list
ExM-user at lists.mcs.anl.gov<mailto:ExM-user at lists.mcs.anl.gov>
https://lists.mcs.anl.gov/mailman/listinfo/exm-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/exm-user/attachments/20150421/da2d02e7/attachment-0001.html>