workshop updates (#80)

Removal of some out-of-date stuff, new labs content, updates of existing labs to better reflect current TF API, colab support in notebooks, addition of 'eager' example, dockerfile update.

Author: amygdala

INSTALL.md

@@ -1,230 +1,76 @@
 
-**Note**: you do not need to follow these instructions for the [Strata workshop](https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/60847) or the [Strange Loop](https://www.thestrangeloop.com) workshop.  For those workshops, you just need [TensorFlow](https://www.tensorflow.org/install/) and [Jupyter notebook](http://jupyter.org/install.html) installed on your laptop.
+# Installation
 
-# Installation instructions for the TensorFlow/Cloud ML workshop
+We suggest installing the necessary libraries locally on your laptop as described below in "__Local Installation__", as that will let you run TensorBoard as part of the workshop, and let you run python scripts as well as notebooks. 
 
-  - [Project and Cloud ML setup](#project-and-cloud-ml-setup)
-  - [Docker-based installation](#docker-based-installation)
-    - [Download the container image](#download-the-container-image)
-    - [Create a directory to hold data files needed by the workshop](#create-a-directory-to-hold-data-files-needed-by-the-workshop)
-    - [Run the container](#run-the-container)
-    - [Authorize gcloud in the running container](#authorize-gcloud-in-the-running-container)
-    - [Restarting the container later](#restarting-the-container-later)
-    - [Running the Docker container on a VM](#running-the-docker-container-on-a-vm)
-  - [Virtual environment-based installation](#virtual-environment-based-installation)
-    - [Install Conda + Python 2.7 to use as your local virtual environment](#install-conda--python-27-to-use-as-your-local-virtual-environment)
-    - [Install TensorFlow into a virtual environment](#install-tensorflow-into-a-virtual-environment)
-    - [Install some Python packages](#install-some-python-packages)
-    - [Install the Google Cloud SDK](#install-the-google-cloud-sdk)
-    - [Cloud ML SDK installation (for 'transfer learning' preprocessing)](#cloud-ml-sdk-installation-for-transfer-learning-preprocessing)
-  - [Set up some data files used in the examples](#set-up-some-data-files-used-in-the-examples)
-    - [Transfer learning example](#transfer-learning-example)
-  - [Optional: Clone/Download the TensorFlow repo from GitHub](#optional-clonedownload-the-tensorflow-repo-from-github)
+But if you have trouble with your installation, you will be able to run everything but TensorBoard in [Colab notebooks instead](https://colab.research.google.com/).
 
-* * *
+## Local Installation
 
-You can set up for the workshop in two different, mutually-exclusive ways:
+It's highly recommended to use a Python virtual environment for this workshop, either [Virtualenv](https://virtualenv.pypa.io/en/stable) or [Conda](http://conda.pydata.org/miniconda.html).  Either Python 2.7 or Python 3 will work.
 
-- [Running in a docker container](#docker-based-installation).
-- [Installing the necessary packages into a virtual environment](#virtual-environment-based-installation).
+Then install the following into your virtual environment:
 
-You will need to do the [Cloud Project and Cloud ML setup](#project-and-cloud-ml-setup) in either case.
+-  Install [TensorFlow](https://www.tensorflow.org/install). Your TensorFlow version must be >=1.7 to run all the labs. Note that even if you're running a conda virtual environment, use `pip install tensorflow` to install TensorFlow (*not* the `conda install` version, which may not be up to date.)
 
-## Project and Cloud ML setup
+-  Install [Jupyter](https://jupyter.org/install.html). 
 
-Follow the instructions below to create a project, initialize it for Cloud ML, and set up a storage bucket to use for the workshop examples.
+- Optionally, install `matplotlib`. (If you're using a conda virtual environment, use `conda install matplotlib`.)
 
-**Note: Skip the "Setting up your environment" section** of the linked-to page at this step. (If you run in a docker container, it is already done for you).
+### Test your installation
 
-* [Setting Up Your GCP Project](https://cloud.google.com/ml/docs/how-tos/getting-set-up#setting_up_your_google_cloud_project )
-* [Initializing Cloud ML for your project](https://cloud.google.com/ml/docs/how-tos/getting-set-up#initializing_your_product_name_short_project)
-* [Setting up your Cloud Storage Bucket](https://cloud.google.com/ml/docs/how-tos/getting-set-up#setting_up_your_cloud_storage_bucket)
-
-## Docker-based installation
-
-We're providing a [Docker](https://www.docker.com/) container image with all the necessary libraries included, for you to download.
-
-To use it, you'll need to have [Docker installed](https://docs.docker.com/engine/installation/).
-To run some of the examples, you'll likely need to configure it with at least 4GB of memory.
-
-If you like, you can start up a Google Compute Engine (GCE) VM with docker installed, and run the container there.  This is a good option if you don't want to install Docker locally. See the instructions at the end of this section for how to do that.
-
-### Download the container image
-
-Once Docker is installed and running, download the workshop image:
-
-```sh
-$ docker pull gcr.io/google-samples/tf-workshop:v7
-```
-
-[Here's the Dockerfile](https://github.com/amygdala/tensorflow-workshop/tree/master/workshop_image) used to build this image.
-
-### Create a directory to hold data files needed by the workshop
-
-Create a directory (called, say, `workshop-data`) to mount as a volume when you start up the docker container.  You can put data/output in this directory so that it is accessible both within and outside the running container.
-
-### Run the container
-
-Once you've downloaded the container image, you can run it like this:
+Start up a jupyter notebooks server from the command line:
 
 ```sh
-docker run -v `pwd`/workshop-data:/root/tensorflow-workshop-master/workshop-data -it \
-    -p 6006:6006 -p 8888:8888 -p 5000:5000 gcr.io/google-samples/tf-workshop:v7
+jupyter notebook .
 ```
 
-Edit the path to the directory you're mounting as appropriate. The first component of the `-v` arg is the local directory, and the second component is where you want to mount it in your running container.
+Create a new notebook and paste the following python code into a notebook cell:
 
-### Authorize gcloud in the running container
+```python
+import tensorflow as tf
 
-In the docker container, run the following commands, replacing `<your-project-name` with your project name.
-For the second two, you will get a URL to paste in your browser, to obtain an auth code. In the browser, allow access, then copy the resulting code and paste it at the prompt in your running docker container.
-
-```shell
-gcloud config set project <your-project-name>
-gcloud auth login
-gcloud auth application-default login
-```
-
-### Restarting the container later
-
-If you later exit your container and then want to restart it again, you can find the container ID by running:
-
-```sh
-$ docker ps -a
-```
-
-Then, run:
-
-```sh
-$ docker start <container_id>
+print(tf.__version__)
 ```
 
-(`docker ps` should then show it running). Once the workshop container is running again, you can exec back into it like this:
+'Run' the cell. Ensure that there are no import errors and that the TensorFlow version is as expected.
 
-```sh
-$ docker exec -it <container_id> bash
-```
+## Using Colaboratory
 
-### Running the Docker container on a VM
+Instead of running in a local installation, all of the jupyter notebooks in this workshop can also be run on [Colab](https://colab.research.google.com/), with the exception of the lab sections that use TensorBoard. 
 
-It is easy to set up a Google Compute Engine (GCE) VM on which to run the Docker container. We sketch the steps below.
+The lab section READMEs include links to launch the workshop notebooks in colab.
 
-First, make sure that your project has the GCE API enabled. An easy way to do this is to go to the [Cloud Console](https://console.cloud.google.com/), and visit the Compute Engine panel.  It should display a button to enable the API.
 
-- Connect to your project's [Cloud Shell](https://cloud.google.com/shell/).
+## Google Cloud Platform
 
-- From the Cloud Shell, create a container-optimized image as described [here](https://cloud.google.com/container-optimized-os/docs/how-to/create-configure-instance). E.g.:
-
-```
-gcloud compute instances create mlworkshop \
-    --image-family gci-stable \
-    --image-project google-containers \
-    --zone us-central1-b --boot-disk-size=100GB \
-    --machine-type n1-standard-1
-```
-
-- Set up a default network firewall rule for: tcp:8888, tcp:6006, and tcp:5000.  (This opens the ports we'll use for jupyter, Tensorboard, and Flask.) E.g:
-
-```shell
-gcloud compute firewall-rules create mlworkshop --allow tcp:8888,tcp:6006,tcp:5000
-```
+Some of the labs include instructions for optionally running training and prediction on Google Cloud Platform (GCP) [Cloud ML Engine](https://cloud.google.com/ml-engine).  
 
-- SSH to your image.  You can do this from the Cloud Console by visiting the Compute Engine panel, and clicking on the 'SSH' pulldown to the right of your instance.
+If you want to play along, set up a GCP account: click the **Try it free** button at the top of this page:
+[cloud.google.com](https://cloud.google.com/).
 
-Then, once you've ssh'd to the VM, follow the instructions above to download and run the Docker container there.
-Note: **Docker is already installed** on the 'container-optimized' VMs.
 
-When you're done with the workshop, you may want to *stop* or *delete* your VM instance.
+## Docker image
 
-You can also delete your firewall rule:
+A Docker image is also available.
 
-```shell
-gcloud compute firewall-rules delete mlworkshop
-```
-
-* * *
-## Virtual environment-based installation
-
-(These steps are not necessary if you have already completed the instructions for running the Docker image.)
-
-We highly recommend that you use a virtual environment for your TensorFlow installation rather than a direct install onto your machine.  The instructions below walk you through a `conda` install, but a `virtualenv` environment will work as well.
-
-### Install Conda + Python 2.7 to use as your local virtual environment
-
-Anaconda is a Python distribution that includes a large number of standard numeric and scientific computing packages. Anaconda uses a package manager called "conda" that has its own environment system similar to Virtualenv.
-
-Follow the instructions [here](https://www.continuum.io/downloads).  The [miniconda version](http://conda.pydata.org/miniconda.html) should suffice.
-
-### Install TensorFlow into a virtual environment
-
-Follow the instructions [on the TensorFlow site](https://www.tensorflow.org/get_started/os_setup#anaconda_installation) to create a Conda environment with Python 2.7, *activate* it, and then install TensorFlow within it.
-
-**Note**: Install TensorFlow version 0.12, by using the [**pip install steps**](https://www.tensorflow.org/get_started/os_setup#anaconda_installation#using_pip) for conda rather than conda-forge.
-
-If you'd prefer to use virtualenv, see [these instructions](https://www.tensorflow.org/get_started/os_setup#virtualenv_installation) instead.
-
-Remember to activate your environment in all the terminal windows you use during this workshop.
-
-### Install some Python packages
+### Download the container image
 
-With your conda environment activated, install the following packages:
+Once Docker is installed and running, download the workshop image:
 
 ```sh
-$ conda install numpy
-$ conda install scipy
-$ pip install sklearn pillow
-$ conda install matplotlib
-$ conda install jupyter
+docker pull gcr.io/google-samples/tf-workshop:v8
 ```
 
-(If you are using `virtualenv` instead of `conda`, install the packages using the equivalent `pip` commands instead).
-
-
-### Install the Google Cloud (gcloud) SDK
-
-Follow the installation instructions [here](https://cloud.google.com/sdk/downloads), then run:
-
-```
-gcloud components install beta
-```
-
-To get the `gcloud beta ml` commands.
-
-### Cloud ML SDK installation (for 'transfer learning' preprocessing)
-
-The Cloud ML SDK is needed to run the 'preprocessing' stage in the [Cloud ML transfer
-learning](workshop_sections/transfer_learning/cloudml) example. It requires Python 2.7 to install. It's possible to
-skip this part of setup for most of the exercises.
-
-To install the SDK, follow the setup instructions
-[on this page](https://cloud.google.com/ml/docs/how-tos/getting-set-up).
-(Assuming you've followed the instructions above, you will have already done some of these steps. **Install TensorFlow version 0.12** as described in [this section](#install-tensorflow-into-a-virtual-environment), not 0.11)
-
-**Note**: if you have issues with the pip install of `python-snappy`, and are running in a conda virtual environment, try `conda install python-snappy` instead.
-
-You don't need to download the Cloud ML samples or docs for this workshop, though you may find it useful to grab them
-anyway.
-
-
-## Set up some data files used in the examples
-
-### Transfer learning example
+[Here's the Dockerfile](https://github.com/amygdala/tensorflow-workshop/tree/master/workshop_image) used to build this image.
 
-Because we have limited workshop time, we've saved a set of
-[TFRecords]([TFRecords](https://www.tensorflow.org/api_docs/python/python_io/))
-generated as part of the [Cloud ML transfer learning](workshop_sections/transfer_learning/cloudml)
-example. To save time, copy them now to your own bucket as follows.
+### Run the container
 
-Set the `BUCKET` variable to point to your GCS bucket (replacing `your-bucket-name` with the actual name), then copy the records to your bucket.  Then, set the GCS_PATH variable to the newly copied subdir.
+Once you've downloaded the container image, run it like this:
 
-```shell
-BUCKET=gs://your-bucket-name
-gsutil cp -r gs://tf-ml-workshop/transfer_learning/hugs_preproc_tfrecords $BUCKET
-GCS_PATH=$BUCKET/hugs_preproc_tfrecords
+```sh
+docker run -it \
+    -p 6006:6006 -p 8888:8888 -p 5000:5000 gcr.io/google-samples/tf-workshop:v8
 ```
 
 
-## Optional: Clone/Download the TensorFlow repo from GitHub
-
-We'll be looking at some examples based on code in the tensorflow repo. While it's not necessary, you might want to clone or download it [here](https://github.com/tensorflow/tensorflow), or grab the latest release [here](https://github.com/tensorflow/tensorflow/releases).
-

README.md

@@ -1,10 +1,9 @@
 
 # TensorFlow workshop materials
 
-This repo contains materials for use in a TensorFlow workshop.  
-Contributions are not currently accepted.  This is not an official Google product.
+This repo contains materials for use in TensorFlow workshops.
 
-This document points to more information for each workshop lab.  
+Contributions are not currently accepted.  This is not an official Google product.
 
 <!---
 [** add note re: TF versions tested with **].
@@ -20,21 +19,21 @@ This document points to more information for each workshop lab.
 
 - [Introducing MNIST, and building a simple linear classifier in TensorFlow](workshop_sections/mnist_series/01_README_mnist_simple.md).
 - [Using TensorFlow's high-level APIs to build an MNIST DNN Classifier, and introducing TensorBoard](workshop_sections/mnist_series/02_README_mnist_tflearn.md).
-- [Building custom `Estimator`s for a version of MNIST that uses CNNs](workshop_sections/mnist_series/mnist_cnn_custom_estimator/README.md), using either TensorFlow or [Keras](https://keras.io/) layers.
+- [An example of using Keras with TensorFlow eager mode, on the'Fashion MNIST' dataset](workshop_sections/mnist_series/mnist_eager_keras.ipynb): This notebook shows an example of using Keras with TensorFlow eager mode, on the 'Fashion MNIST' dataset. This notebook requires TensorFlow >=1.7.
+- [Building Custom `Estimator`s for a version of MNIST that uses CNNs](workshop_sections/mnist_series/mnist_cnn_custom_estimator/README.md), using either TensorFlow or [Keras](https://keras.io/) layers.
 
 
 ## 'Wide & Deep'
 
-- [Using a tf.estimator to train a 'Wide & Deep' model](workshop_sections/wide_n_deep/README.md).
+- [Using a tf.estimator to train a 'Wide & Deep' model](workshop_sections/wide_n_deep/README.md). The example highlights use of [`tf.feature_columns`](https://www.tensorflow.org/versions/master/get_started/feature_columns), which are intermediaries between raw data and Estimators, enabling you to transform a diverse range of raw data into formats that Estimators can use, and allowing easy experimentation.
+It also includes the use of [**`tf.estimator.train_and_evaluate`**](https://www.tensorflow.org/api_docs/python/tf/estimator/train_and_evaluate) and [**Datasets**](https://www.tensorflow.org/api_docs/python/tf/data/Dataset).
 
-## Transfer Learning
 
-- [Transfer learning: using a trained model to 'bootstrap' learning new classifications](workshop_sections/transfer_learning/README.md).
-    + [Using Cloud ML](workshop_sections/transfer_learning/cloudml)
-    + **(possibly outdated)** [Using a custom Estimator](workshop_sections/transfer_learning/TF_Estimator)
+## Extras
 
-- **(possibly outdated)** [Building a word2vec model using a Custom Estimator, and exploring the learned embeddings](workshop_sections/word2vec/README.md). Introducing [TFRecords](https://www.tensorflow.org/api_guides/python/python_io).
+- [Transfer learning: using a trained model to 'bootstrap' learning new classifications](transfer_learning/README.md) [using Cloud ML Engine](workshop_sections/transfer_learning/cloudml). This example still works (and is fun to play with), but uses TensorFlow v1.2.  So it doesn't demonstrate current best practices.
 
-## Extras
+- **(probably outdated)** [Building a word2vec model using a Custom Estimator, and exploring the learned embeddings](workshop_sections/word2vec/README.md). Introducing [TFRecords](https://www.tensorflow.org/api_guides/python/python_io).
+
+In addition, there is an [extras](workshop_sections/extras/README.md) directory, that contains some older labs not currently used in this workshop (& which may not necessarily run with the latest version of TF), but which may be of interest.
 
-In addition, there is an [extras](workshop_sections/extras/README.md) directory, that contains some older labs not currently used in this workshop (and which may not necessarily run with the latest version of TF), but which may be of interest.

requirements.txt

@@ -0,0 +1,5 @@
+tensorflow>=1.7
+matplotlib
+numpy
+pandas
+jupyter

workshop_image/Dockerfile

@@ -11,33 +11,36 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-FROM gcr.io/tensorflow/tensorflow:latest-devel
+FROM gcr.io/tensorflow/tensorflow:latest
 
 RUN pip install --upgrade pip
 RUN apt-get update
 RUN apt-get install -y unzip python-dev python-pip zlib1g-dev libjpeg-dev libblas-dev
 RUN apt-get install -y liblapack-dev libatlas-base-dev libsnappy-dev libyaml-dev gfortran
 RUN apt-get install -y python-scipy
 
-RUN pip install sklearn nltk pillow setuptools
+RUN pip install sklearn pillow setuptools jupyter
 RUN pip install flask google-api-python-client
 RUN pip install pandas python-snappy scipy scikit-learn requests uritemplate
-# RUN pip install --upgrade --force-reinstall https://storage.googleapis.com/cloud-ml/sdk/cloudml.latest.tar.gz
+RUN python -mpip install matplotlib
 
+# RUN pip install nltk
 # RUN python -c "import nltk; nltk.download('punkt')"
 
-RUN curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-171.0.0-linux-x86_64.tar.gz | tar xvz
+RUN curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-198.0.0-linux-x86_64.tar.gz | tar xvz
 RUN ./google-cloud-sdk/install.sh -q
-# RUN ./google-cloud-sdk/bin/gcloud components install beta
 
 ADD download_git_repo.py download_git_repo.py
 ENV PATH="${PATH}:/root/google-cloud-sdk/bin"
 
+ADD start_notebook.sh start_notebook.sh
+RUN chmod +x start_notebook.sh
+
 # TensorBoard
 EXPOSE 6006
-# IPython
+# For jupyter
 EXPOSE 8888
 # Flask
 EXPOSE 5000
 
-CMD ["sh", "-c", "python download_git_repo.py ; /bin/bash"]
+CMD ["sh", "-c", "python download_git_repo.py; ./start_notebook.sh ; /bin/bash"]

workshop_image/start_notebook.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+jupyter notebook --allow-root &> /dev/null &