masonry should set PYTHONPATH (or equiv) when making manuals

[markcmiller86]

Describe the bug

I was trying to do a build with masonry from a partially built state. When I did, I encountered this error during the manuals_release phase (which is trying to run sphinx_build to make the manuals)...

Traceback (most recent call last):
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
ws.require(__requires__)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (Sphinx 5.0.2 (/Users/miller86/Library/Python/3.7/lib/python/site-packages), Requirement.parse('Sphinx==2.2.1'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/sphinx-build", line 6, in 
from pkg_resources import load_entry_point
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in 
@_call_aside
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
f(*args, **kwargs)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
return cls._build_from_requirements(__requires__)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'Sphinx==2.2.1' distribution was not found and is required by the application
make[3]: *** [doc/CMakeFiles/manuals] Error 1
make[2]: *** [doc/CMakeFiles/manuals.dir/all] Error 2
make[1]: *** [doc/CMakeFiles/manuals.dir/rule] Error 2
make: *** [manuals] Error 2

Its a lengthy error message but the key point is that it winds up loading pkg_release from a wholly different python3 installation...the one that is part of Xcode. We've probably gotten lucky in the past that this didn't bite us somehow but to resolve, before we run sphinx_build we probably need to set PYTHONPATH to ensure python winds up getting all its modules from VisIt's python installation.

[cyrush]

I don't think that alone would fix it - if its still using the other python, even if PYTHONPATH is set, you could get very strange results.

Make manuals should be using the python passed to visit, do we know everything is ok there?

[cyrush]

Just to double check, --- Is masonry using VisIt's build system to make the manuals? (Calling our cmake targets) or is it doing them as a separate command?

[markcmiller86]

Is masonry using VisIt's build system to make the manuals? (Calling our cmake targets) or is it doing them as a separate command?

haven't investigated that yet but I am not sure it matters. Either way would ultimately result in a sphinx-build command being invoked (either directly from masonry or indirectly from masonry calling make manuals and the makefiles running sphinx-build), right?

could the problem be that pkg_release winds up being an egg in our Python install? No matter what I do with the python installed with masonry for VisIt, I can't get it to find pkg_release there. It always resolves to the one in Xcode.

[cyrush]

ideally: make manual calls sphinx-build that exists inside of our python install. B/c of this it should be fully independent from the python used by masonry to bootstrap tpls and build visit.

[cyrush]

It is calling make manuals:

visit/src/tools/dev/masonry/bootstrap_visit.py

Line 215 in 05a7248

def steps_manuals(opts,build_type,ctx):

Grumble, something seems off.

[cyrush]

do you have PYTHONPATH set in your env?

[markcmiller86]

do you have PYTHONPATH set in your env?

Nope. env | grep -i py produces nothing.

Yes, it is calling make manuals which, after I threaded through the CMake nuggets, the actual command executed looks like...

cd /Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/build.release/doc && \ 
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/sphinx-build -b html \
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/visit/src/doc /Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/build.release/resources/help/en_US/manual -a

And this fails with...

Traceback (most recent call last):
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (Sphinx 5.0.2 (/Users/miller86/Library/Python/3.7/lib/python/site-packages), Requirement.parse('Sphinx==2.2.1'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/sphinx-build", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'Sphinx==2.2.1' distribution was not found and is required by the application

which is interesting because I do have a sphinx-build in ~/Library/Python/3.7/bin which is version 5.0.2. But that is NOT in my default path and, furthermore, I don't pass my default path with running masonry. I start masonry with env PATH=/usr/local/bin:/usr/bin:/bin:/sbin which is a vanilla path.

When I go examine the sphinx executables installed with VisIt via masonry, I find that they are all sh-banged with absolute path to Xcode.app python3...

find /Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7 -name '*.py' -exec grep 'Xcode.app' {} \; -print
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2xetex.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2latex.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2odt_prepstyles.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2html4.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2html5.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2xml.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2odt.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2pseudoxml.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2s5.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2html.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rst2man.py
#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/rstpep2html.py
            #   /Applications/Xcode.app/Contents/Developer/Platforms/
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/lib/python3.7/distutils/unixccompiler.py
    FRAMEWORKS_PATH = '/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS8.1.sdk/System/Library/Frameworks/'
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/lib/python3.7/site-packages/Pygments-2.5.2-py3.7.egg/pygments/lexers/_cocoa_builtins.py
    #    can only be found inside Xcode.app if the "Command Line Tools"
/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/lib/python3.7/_osx_support.py

[markcmiller86]

Let me be a bit clearer...the search above examins only .py files that were installed to the Python bin directory VisIt created for itself. There are a number of other python scripts there that do not have .py extension and they are all sh-banged with abs path to Xcode.app python instead of this python.

[scratlantis:3.7.7/darwin-x86_64/bin] miller86% grep Xcode.app *
chardetect:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
cygdb:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
cython:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
cythonize:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
easy_install:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
easy_install-3.7:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
f2py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
f2py3:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
f2py3.7:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
pybabel:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
pygmentize:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2html.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2html4.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2html5.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2latex.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2man.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2odt.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2odt_prepstyles.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2pseudoxml.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2s5.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2xetex.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rst2xml.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
rstpep2html.py:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
sphinx-apidoc:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
sphinx-autogen:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
sphinx-build:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3
sphinx-quickstart:#!/Applications/Xcode.app/Contents/Developer/usr/bin/python3

[markcmiller86]

Changing the sh-bang in sphinx-build in the above from #!/Applications/Xcode.app/Contents/Developer/usr/bin/python3 to #!/Users/miller86/visit/visit/33rc/release/build-mb-3.3.0-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/python3 so that it winds up using the python3 installed as part of the VisIt build instead of the python3 in Xcode.app fixes the problem building the manuals.

So, somehow, we need to make sure that these tools all get installed with the proper shbang. I am guessing that somewhere in the build process we wound up using an unqualified python3 (resulting in whatever the path defaults too which for me results in

env PATH=/usr/local/bin:/usr/bin:/bin:/sbin which python3
/usr/bin/python3
env PATH=/usr/local/bin:/usr/bin:/bin:/sbin python3
Python 3.7.3 (default, Nov 15 2019, 04:04:52) 
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python37.zip', \
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7', \
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/lib-dynload', \
'/Users/miller86/Library/Python/3.7/lib/python/site-packages', \
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages', \
'/Library/Python/3.7/site-packages']
>>> 

[cyrush]

I think when VisIt's Python was built, something had to be crossed with the system python.

If i look at my Sphinx build for 3.3, sh-bang has the VisIt built python:

  head build-mb-3.2.2-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/sphinx-build 

#!/Users/harrison37/Work/github/visit-dav/3.2.2_release/visit/build-mb-3.2.2-darwin-10.14-x86_64-release/thirdparty_shared/third_party/python/3.7.7/darwin-x86_64/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'Sphinx==2.2.1','console_scripts','sphinx-build'
__requires__ = 'Sphinx==2.2.1'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])

Culprits are usually python in your path, or PYTHONPATH or even an anaconda install somewhere. But I know you checked all of those things. Mysteries like this are frustrating.

[markcmiller86]

Ok, thanks for looking at the sh-bangs in your case. That is good to know it will get it right under the right conditions.

Well, I ensured $PATH is generic and there is no $PYTHONPATH in my env though I didn't think to explicitly override anything with env PYTHONPATH="" and I do have Anaconda installed but not "visible" to anything from my terminal.

By build completed but a) I accedentially vpn'd in to file TC in the middle of it...codesigning didn't fail but I haven't confirmed it yet and b) I am still finding some .so files in numpy and PIL with paths to /opt/local/lib...

	/opt/local/lib/libfreetype.6.dylib 
	/opt/local/lib/liblcms2.2.dylib 
	/opt/local/lib/libopenblas-r1.dylib 
	/opt/local/lib/libopenjp2.7.dylib 
	/opt/local/lib/libtiff.5.dylib 
	/opt/local/lib/libwebp.7.dylib 
	/opt/local/lib/libwebpdemux.2.dylib 
	/opt/local/lib/libwebpmux.3.dylib 
	/opt/local/lib/libxcb.1.dylib 

I am investigating why this is happening with these two python modules.

[markcmiller86]

@cyrush I got curious and examined the 3.2.2 release. I don't see anything there that depends on blas or openblas. So, I think the numpy build on my system is building some additional features we haven't necessarily supported in the past.

And, I think some of the python modules, like numpy, are kinda sorta designed to go looking for stuff in "special places" like in MacPorts (/opt/local/lib) or in Fink (/sw/include) because I see these paths in lists of paths used as part of the numpy install all over the numpy sources.