Fix develop/editable install mode (horovod#3074)

Signed-off-by: Nicolas Castet <[email protected]>

Author: nvcastet

CHANGELOG.md

@@ -18,6 +18,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ### Fixed
 
+- Fix Horovod develop/editable install mode and incremental builds. ([#3074](https://github.com/horovod/horovod/pull/3074))
+
 ## [v0.22.1] - 2021-06-10
 
 ### Added

docs/contributors.rst

@@ -20,19 +20,14 @@ Develop within a virtual environment to avoid dependency issues:
 
 .. code-block:: bash
 
-    $ virtualenv env
+    $ python3 -m venv env
     $ . env/bin/activate
 
 We recommend installing package versions that match with those under test in
 `Buildkite <https://github.com/horovod/horovod/blob/master/.buildkite/gen-pipeline.sh>`__.
 The following versions are recommended (see default versions defined through :code:`ARG` in
 `Dockerfile.test.cpu <https://github.com/horovod/horovod/blob/master/Dockerfile.test.cpu>`__ and
 `Dockerfile.test.gpu <https://github.com/horovod/horovod/blob/master/Dockerfile.test.gpu>`__ file.
-The versions with CPU support can be installed through the provided :code:`setup.py` file:
-
-.. code-block:: bash
-
-    pip install -e .[dev,test]
 
 You can find all other non-Python packages that need to be installed on your system for Horovod to build
 in the `Dockerfile.test.cpu <https://github.com/horovod/horovod/blob/master/Dockerfile.test.cpu>`__ and
@@ -42,30 +37,27 @@ Specifically, see all :code:`RUN apt-get install` lines.
 Build and Install
 -----------------
 
-First, uninstall any existing version of Horovod.  Be sure to do this *outside* the Horovod root directory:
+From *inside* the Horovod root directory, install Horovod in develop/editable mode:
 
 .. code-block:: bash
 
-    $ cd $HOME
-    $ pip uninstall -y horovod
-    $ cd -
-
-From *inside* the Horovod root directory, remove any previous build artifacts and then install Horovod:
-
-.. code-block:: bash
-
-    $ rm -rf build/ dist/
-    $ HOROVOD_WITH_PYTORCH=1 HOROVOD_WITH_TENSORFLOW=1 python setup.py install
+    $ HOROVOD_WITH_PYTORCH=1 HOROVOD_WITH_TENSORFLOW=1 pip install -v -e .
 
 Set ``HOROVOD_WITHOUT_[FRAMEWORK]=1`` to disable building Horovod plugins for that framework.
 This is useful when you’re testing a feature of one framework in particular and wish to save time.
 
-For a debug build with checked assertions etc. replace the invocation of setup.py by:
+Set ``HOROVOD_WITH_[FRAMEWORK]=1`` to generate an error if the Horovod plugin for that framework failed to build.
 
-.. code-block:: bash
+Set ``HOROVOD_DEBUG=1`` for a debug build with checked assertions, disabled compiler optimizations etc.
+
+Other environmental variables can be found in the `install documentation <https://github.com/horovod/horovod/blob/master/docs/install.rst#environment-variables>`__.
 
-    $ HOROVOD_WITH_PYTORCH=1 HOROVOD_WITH_TENSORFLOW=1 python setup.py build_ext --debug install
+You can install optional dependencies defined in `setup.py <https://github.com/horovod/horovod/blob/master/setup.py>`__ by adding brackets
+at the end of the command line e.g. ``[test]`` for test dependencies.
+If you have not installed specific DL frameworks yet, add ``[dev]`` to install the CPU version of all supported DL frameworks.
 
+In develop mode, you can edit the Horovod source directly in the repo folder. For Python code, the changes will take effect
+immediately. For **C++/CUDA code**, the ``... pip install -v -e .`` command needs to be invoked again to perform an incremental build.
 
 Testing
 -------

docs/install.rst

@@ -226,6 +226,7 @@ Optional environment variables that can be set to configure the installation pro
 
 Possible values are given in curly brackets: {}.
 
+* ``HOROVOD_DEBUG`` - {1}. Install a debug build of Horovod with checked assertions, disabled compiler optimizations etc.
 * ``HOROVOD_BUILD_ARCH_FLAGS`` - additional C++ compilation flags to pass in for your build architecture.
 * ``HOROVOD_CUDA_HOME`` - path where CUDA include and lib directories can be found.
 * ``HOROVOD_BUILD_CUDA_CC_LIST`` - List of compute capabilities to build Horovod CUDA kernels for (example: ``HOROVOD_BUILD_CUDA_CC_LIST=60,70,75``)

setup.py

@@ -16,6 +16,7 @@
 # ==============================================================================
 
 import os
+import shutil
 import subprocess
 import sys
 import textwrap
@@ -25,6 +26,7 @@
 
 from horovod import __version__
 
+_FRAMEWORK_METADATA_FILE = 'horovod/metadata.json'
 
 class CMakeExtension(Extension):
     def __init__(self, name, cmake_lists_dir='.', sources=[], **kwa):
@@ -52,6 +54,8 @@ def is_build_action():
     if sys.argv[1].startswith('install'):
         return True
 
+    if sys.argv[1].startswith('develop'):
+        return True
 
 def get_cmake_bin():
     return os.environ.get('HOROVOD_CMAKE', 'cmake')
@@ -66,7 +70,7 @@ def build_extensions(self):
 
         cmake_bin = get_cmake_bin()
 
-        config = 'Debug' if self.debug else 'RelWithDebInfo'
+        config = 'Debug' if self.debug or os.environ.get('HOROVOD_DEBUG') == "1" else 'RelWithDebInfo'
 
         ext_name = self.extensions[0].name
         build_dir = self.get_ext_fullpath(ext_name).replace(self.get_ext_filename(ext_name), '')
@@ -98,6 +102,13 @@ def build_extensions(self):
         except OSError as e:
             raise RuntimeError('CMake failed: {}'.format(str(e)))
 
+        if sys.argv[1].startswith('develop'):
+            # Copy over metadata.json file from build directory
+            shutil.copyfile(os.path.join(build_dir, _FRAMEWORK_METADATA_FILE),
+                            os.path.join(self.extensions[0].cmake_lists_dir, _FRAMEWORK_METADATA_FILE))
+            # Remove unfound frameworks, otherwise develop mode will fail the install
+            self.extensions = [x for x in self.extensions if os.path.exists(self.get_ext_fullpath(x.name))]
+
 
 # python packages required to use horovod in general
 require_list = ['cloudpickle', 'psutil', 'pyyaml', 'dataclasses;python_version<"3.7"']