Py3 and tensorflow setup

iNLyze · February 23, 2017, 10:56pm

Thanks, I’ll consider it. Especially the part on speed might be interesting. I’ve had some speed issues occasionally with part I.

Even · February 24, 2017, 12:12am

+1 for Anaconda. Not only is it a separate environment, you can actually create different setups within it that have specific libraries imported. I wasn’t sure about switching to it either, but after doing so for this course I’m never going back.

Jonas · February 24, 2017, 7:14pm

It works!

Because I’m using spot instances which I set up and tear down every time, I updated my setup script to reflect the above changes:
https://github.com/jonas-pettersson/fast-ai/blob/master/scripts/install-gpu-tf.sh

Here is the description how to set up an AWS spot instance using the script:

github.com

jonas-pettersson/fast-ai/blob/master/README.md

# fast-ai  
http://course.fast.ai  
## Script to automatically setup an AWS spot instance and mount with existing volume 
`git clone https://github.com/jonas-pettersson/fast-ai`  
`cd fast-ai/scripts/`  

Please carefully read the comments in [setup_aws_spot_w_remount.sh](https://raw.githubusercontent.com/jonas-pettersson/fast-ai/master/scripts/setup_aws_spot_w_remount.sh)  
Then you are ready to run:   
`. setup_aws_spot_w_remount.sh` 

## Script to automatically remove the AWS spot instance
Please carefully read the comments in [remove_aws_spot.sh](https://raw.githubusercontent.com/jonas-pettersson/fast-ai/master/scripts/remove_aws_spot.sh)  
Then you are ready to run:   
`./remove_aws_spot.sh` 

## Manual process to setup spot instance from scratch 

**1) Request Spot Instance**  
AWS Console -> (Login) -> EC2 Dashboard -> Spot Requests
"Request Spot Instances"

This file has been truncated. show original

jeremy · February 24, 2017, 7:52pm

For spot instances, would it be easier to have Anaconda etc installed onto an EBS volume, and then simply attach the volume to the new instance after you create it? (Which you could do in your script). For instance, you could attach the EBS volume as your home directory, which means you’ll also have your configuration changes saved automatically.

Jonas · February 24, 2017, 8:20pm

thanks, I will try that. I believe I still pay something for a volume per GiB even when it is not attached to an instance. but it is not so much. I will try and let you know.
if I can save the time for file transfer it might be worth it.

UPDATE: hmm… cannot detach a volume from an instance while it is running, and it is a feature of the spot instance that it cannot be stopped, just terminated. With termination the volume is gone, of course, so it seems I’m stuck with the procedure of setting it up again. Which is fine, at least for me.

UPDATE 2: I was wrong in my belief that volume from spot instance cannot be saved - there is an option to that (“DeleteOnTermination”: false,). Inspired by @slavivanov I am now working on a similar approach. However I think it is not necessary to generate JSON from text, instead I can use a JSON conf file like this:
aws ec2 request-spot-fleet --spot-fleet-request-config file://config.json
I will work on this and update when I have it ready.

jeremy · February 24, 2017, 9:52pm

This looks like a great approach: Persistent AWS Spot Instances (How to)

jpuderer · February 25, 2017, 1:36am

Hey everybody!

I’ve been working on migrating some of the models (and pre-trained weights) from Part 1 of this course to TensorFlow. This is mostly so I can export TensorFlow graphs that will run on mobile devices (TensorFlow has decent support for running trained models on mobile devices).

I’ve put together a Python Notebook that walks through the conversion process and some of the gotchas:

github.com

jpuderer/cats_and_dogs_using_tensorflow/blob/master/cats_and_dogs_using_tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cats and Dogs using the TensorFlow Backend\n",
    "\n",
    "This notebook shows how to take the 'Cats and Dogs' classifier from the very first lesson of the amazing 'Practical Deep Learning For Coders' course, and convert it to run using TensorFlow (instead of Theano).\n",
    "\n",
    "http://course.fast.ai/\n",
    "\n",
    "Why do we care what backend we use?\n",
    "\n",
    "There are a few reasons, many of which are well covered here (as well as some good instructions on installing TensorFlow):\n",
    "    http://www.pyimagesearch.com/2016/11/14/installing-keras-with-tensorflow-backend/\n",
    "\n",
    "My compeling use case is that TensorFlow has good support (and example code) for running models on Android and iOS.  However, in order to use them, we'll need to be able to export our Keras model as a TensorFlow graph (more on this later).\n",
    "\n",
    "For simplicity (and so you can inspect every step), I've taken the examples from the course notebook, and modified them so that everything fits into a single notebook without external library files."

This file has been truncated. show original

It might be useful to some of you. Especially, if you already have a trained model that you want to use with TensorFlow.

Cheers!
James

Update (Feb 27th): Sadly, something is not right in my script. Everything works when using Theano dim ordering, but the conversion to TensorFlow dim ordering is broken. Somewhere… I updated the notebook to make things clearer. Could use some help, if anyone else is interested.

Jonas · February 25, 2017, 10:33am

Thanks again, I will have a closer look and report any results

UPDATE: it works very well, you can create a new spot instance, mount an existing volume and use it in the way @jeremy proposed. I will finalize, test and document the following scripts, but I post it so anyone interested can already have a look to see how I did it.

setup_aws_spot_w_remount.sh: sets up a new spot instance and executes the remount
specification.json: configuration file for creating the spot instance
remount_root.sh: remount script executed on the newly created spot instance
remove_aws_spot.sh: cancel spot request, terminate instance, and remove the (empty) default volume of the spot instance

UPDATE 2: Above scripts are now tested, corrected, and documented. For sure not everything is checked and, because of the swap root volume operation, should be used with some care. But they would anyhow be useful for someone using the approach of spot instances and mounting an existing volume to root. Further explanations directly in the scripts.

sakiran · February 25, 2017, 1:57pm

After upgrading, will I be able to run the scripts from part-1?

jpuderer · February 25, 2017, 2:04pm

@sakiran It depends. The Vgg model from Part 1 doesn’t work without some modifications, and the any saved weights for convolutional layers needs some significant adjustments to work.

See my post/notebook above for more details.

sakiran · February 25, 2017, 2:08pm

There wouldn’t be any issue if I have python-2 and python-3 simultaneously. Correct?

jpuderer · February 25, 2017, 2:34pm

@sakiran Python3 will be your default environment after upgrading, you won’t be using both simultaneously (it uses whichever version is in your path, in this case Python3).

You shouldn’t have any issues, other than a few particularities to Python3 syntax that should be simple to fix when you encounter them.

Even · February 25, 2017, 7:39pm

@jpuderer are you sure it’s working correctly? The training of the model results in a validation accuracy of 0.5000 aka random chance for a 2 class system. It doesn’t seem like the model is being optimized properly unless I’m missing something.

jpuderer · February 25, 2017, 8:02pm

@even Pretty sure. It’s just that I included the sample set (which is too small to train with, but the full set is too large for Github). I used the full data set, and it trains to the same level as Theanos.

I should probably update the notebook with the output from training on the full set, since if you don’t see the small comment, it’s easy to be misled.

jpuderer · February 25, 2017, 8:09pm

I’ll post the results in a bit. It will be a little while, I’m using a CPU at the moment.

Edit (Feb 26th): Something is not quite right. It trains, but not as well as the original Theano network. I must have made a mistake somewhere. I’ll post an update when I know what it is.

Edit (Feb 27th): Something is not right in my script. Everything works when using Theano dim ordering, but the conversion to TensorFlow dim ordering is broken. Somewhere… Could use some help.

Kjeanclaude · February 25, 2017, 8:14pm

Thank you @jpuderer for the good job !
I have just read what changes to do in the github.
But, please could you confirm that with your changes in the github link above, we could run both part 1 and part 2 scripts ?

jeremy · February 25, 2017, 8:24pm

We’ll be using a different approach to using vgg and resnet in part 2, so you’ll probably want to make some changes to your scripts based on them. (The model weights in part 1 don’t work in tensorflow, and theano weight files you’ve saved won’t work with tensorflow either.)

Other than that, just about everything should work fine. You can always switch backends by changing your keras config file, of course.

jpuderer · February 25, 2017, 8:27pm

@Kjeanclaude You would need to maintain two separate sets of weights for running under Theano or TensorFlow, or be constantly converting between the two regularly.

It probably isn’t worth the trouble. I would just pick one (ie. TensorFlow for this course), convert your weights once, and stick with that.

Kjeanclaude · February 25, 2017, 8:39pm

OK, good to heard that.
Indeed I was writing something to allow those who want to run the part 1 scripts locally to do it via a docker image. But with your work, I think we would have more advantage to make the necessary conversions and then run part 1 and part 2 scripts on the same installation (through a VM and virtual gpu for local use).
Great job !

mariya · February 26, 2017, 2:50am

Since I didn’t take Part One with you and am still catching up on the MOOC and associated homeworks, it would be ideal to keep the Theano / Python 2 environment separate from the Tensorflow / Python 3 until I develop sufficient facility to port over the older code. What would be the easiest way to create a separate AMI for the new setup so I can continue chugging along on the old stuff?