Lesson 8 homework assignments

Part 2 (2017)
56

Couple things you can do (posted from phone and not proof read so caveat emptor):

skimage.color.gray2rgb - will work with normal VGG. There are similar functions in PIL and opencv if you’re using those.

np.repeat(np.expand_dims(img, -1), 3, axis=-1) will work too.

Or if you’re going to be working with a lot of grayscale images, you can modify the VGG model. If you sum the weights (+/- biases) along the channel axis of the first conv layer you’ll get the weights for grayscale VGG.

w, = first_conv_layer.get_weights().sum(axis=-1)

57

Why are we dividing the style image size by 3.5 and in the simpson style image we are dividing it by 2.7? @jeremy

58

From what I remember, the original Starry Night image was pretty big. Jeremy divided by some number (3.5) that brought it down in size to something closer to the content image. I’m guessing the Simpson style image was also pretty big, but not quite as big, and so it was only divided by 2.7. A heuristic is probably to get the style image close to the resolution of your content image (although, it might be a fun experiment to use style images with very different resolutions than the content image).

1 Like
59

Hi @resdntalien, I think the random image that we start with has to be of same size as content image.Because content loss requires both of them to be same shape.But in case of style loss, the random image need not have the same size as style image as long as they are considered from the same layer.This is because we are using gram matrix which will have dimensions of num_filtersxnum_filters

60

@jeremy

Hi Jeremy, I’m going through the MOOC right now and I’m getting a similar issue but when I remove the /100 it still doesn’t work. My code is here: https://github.com/zaoyang/fast_ai_course/tree/master/part2_orig
https://github.com/zaoyang/fast_ai_course/blob/master/part2_orig/neural-style.ipynb

The only thing I have different is the requirements with Keras and I did some minor tweaks to make it work with the latest Keras. Any idea what’s wrong?

In requirements.txt, it has my environment. I’m tried this locally on Mac and on the AWS server with Ubuntu.

Conda environment:

PIP environment:

Both get this yellow image issues. Is this an environment issue or code issue?

61

@jeremy The other issues I’m running into is the following. This is happening on both Ubuntu and Mac as well. I didn’t change anything. So I’m not sure if this is an issue with changing from Theano to TF or another issues. Online it says that it’s bad initalizations of shapes but not sure what the issue could be.

indent preformatted text by 4 spaces
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1038     try:
-> 1039       return fn(*args)
   1040     except errors.OpError as e:

~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
   1020                                  feed_dict, fetch_list, target_list,
-> 1021                                  status, run_metadata)
   1022 

~/anaconda3/envs/ml_py3/lib/python3.6/contextlib.py in __exit__(self, type, value, traceback)
     88             try:
---> 89                 next(self.gen)
     90             except StopIteration:

~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py in raise_exception_on_not_ok_status()
    465           compat.as_text(pywrap_tensorflow.TF_Message(status)),
--> 466           pywrap_tensorflow.TF_GetCode(status))
    467   finally:

InvalidArgumentError: Incompatible shapes: [64] vs. [128]
	 [[Node: gradients_1/add_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _class=["loc:@add_1"], _device="/job:localhost/replica:0/task:0/cpu:0"](gradients_1/add_1_grad/Shape, gradients_1/add_1_grad/Shape_1)]]

During handling of the above exception, another exception occurred:

InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-71-d733ab4b440a> in <module>()
----> 1 x = solve_image(evaluator, iterations, x)

<ipython-input-27-99c5991ad031> in solve_image(eval_obj, niter, x)
      2     for i in range(niter):
      3         x, min_val, info = fmin_l_bfgs_b(eval_obj.loss, x.flatten(),
----> 4                                          fprime=eval_obj.grads, maxfun=20)
      5         x = np.clip(x, -127,127)
      6         print('Current loss value:', min_val)

~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py in fmin_l_bfgs_b(func, x0, fprime, args, approx_grad, bounds, m, factr, pgtol, epsilon, iprint, maxfun, maxiter, disp, callback, maxls)
    191 
    192     res = _minimize_lbfgsb(fun, x0, args=args, jac=jac, bounds=bounds,
--> 193                            **opts)
    194     d = {'grad': res['jac'],
    195          'task': res['message'],

~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py in _minimize_lbfgsb(fun, x0, args, jac, bounds, disp, maxcor, ftol, gtol, eps, maxfun, maxiter, iprint, callback, maxls, **unknown_options)
    326             # until the completion of the current minimization iteration.
    327             # Overwrite f and g:
--> 328             f, g = func_and_grad(x)
    329         elif task_str.startswith(b'NEW_X'):
    330             # new iteration

~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py in func_and_grad(x)
    276     else:
    277         def func_and_grad(x):
--> 278             f = fun(x, *args)
    279             g = jac(x, *args)
    280             return f, g

~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/scipy/optimize/optimize.py in function_wrapper(*wrapper_args)
    290     def function_wrapper(*wrapper_args):
    291         ncalls[0] += 1
--> 292         return function(*(wrapper_args + args))
    293 
    294     return ncalls, function_wrapper

<ipython-input-25-95afa49bd58f> in loss(self, x)
      3 
      4     def loss(self, x):
----> 5         loss_, self.grad_values = self.f([x.reshape(self.shp)])
      6         return loss_.astype(np.float64)
      7 

~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py in __call__(self, inputs)
   2227         session = get_session()
   2228         updated = session.run(self.outputs + [self.updates_op],
-> 2229                               feed_dict=feed_dict)
   2230         return updated[:len(self.outputs)]
   2231 

~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    776     try:
    777       result = self._run(None, fetches, feed_dict, options_ptr,
--> 778                          run_metadata_ptr)
    779       if run_metadata:
    780         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
    980     if final_fetches or final_targets:
    981       results = self._do_run(handle, final_targets, final_fetches,
--> 982                              feed_dict_string, options, run_metadata)
    983     else:
    984       results = []

~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1030     if handle is None:
   1031       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1032                            target_list, options, run_metadata)
   1033     else:
   1034       return self._do_call(_prun_fn, self._session, handle, feed_dict,

~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1050         except KeyError:
   1051           pass
-> 1052       raise type(e)(node_def, op, message)
   1053 
   1054   def _extend_graph(self):

InvalidArgumentError: Incompatible shapes: [64] vs. [128]
	 [[Node: gradients_1/add_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _class=["loc:@add_1"], _device="/job:localhost/replica:0/task:0/cpu:0"](gradients_1/add_1_grad/Shape, gradients_1/add_1_grad/Shape_1)]]

Caused by op 'gradients_1/add_1_grad/BroadcastGradientArgs', defined at:
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 477, in start
    ioloop.IOLoop.instance().start()
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/zmq/eventloop/ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tornado/ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell
    handler(stream, idents, msg)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2698, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2802, in run_ast_nodes
    if self.run_code(code, result):
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-67-de37f8a93ea6>", line 2, in <module>
    grads = K.gradients(loss, model.input)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2264, in gradients
    return tf.gradients(loss, variables, colocate_gradients_with_ops=True)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 560, in gradients
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 368, in _MaybeCompile
    return grad_fn()  # Exit early
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 560, in <lambda>
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py", line 598, in _AddGrad
    rx, ry = gen_array_ops._broadcast_gradient_args(sx, sy)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 411, in _broadcast_gradient_args
    name=name)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

...which was originally created as op 'add_1', defined at:
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
[elided 18 identical lines from previous traceback]
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-67-de37f8a93ea6>", line 1, in <module>
    loss = sum(style_loss(l1[0], l2[0]) for l1,l2 in zip(layers, targs))
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 821, in binary_op_wrapper
    return func(x, y, name=name)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 73, in add
    result = _op_def_lib.apply_op("Add", x=x, y=y, name=name)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [64] vs. [128]
	 [[Node: gradients_1/add_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _class=["loc:@add_1"], _device="/job:localhost/replica:0/task:0/cpu:0"](gradients_1/add_1_grad/Shape, gradients_1/add_1_grad/Shape_1)]]
62

Hi @zaoyang, In Lesson 8 discussions some of us including myself had same problems.Check lesson 8 discussions for fix

1 Like
63

thanks I’ll check it out. Did the Keras 2 implementation from Romano work for you? It wasn’t exactly the same issue. It’s the orange blob.

64

Hi @zaoyang, try , loss = K.mean(metrics.mse(layer, targ)) .I think the current content loss function you have doesn’t return a single value

2 Likes
65

Here:

Here’s my conda and pip environments:


66

Thank you that worked. That removed the orange output. I just don’t understand why that worked.

67

Hi @zaoyang, The loss returned by the original function won’t be a single value if you are using keras 2. So I had to wrap a K.mean around it

1 Like
68

k Thank you appreciate it. I have the invalid argument as well. Not sure if encountered it. I’ll try romano’s code unless you have a quick suggestion.

69

My answers of question 3, anyone who find something wrong or incomplete, please help me correct them, thanks.

1 : What problem are they solving?

They figure out an algorithm which could generate artistic image by deep learning

2 : What is the general idea that they are using to solve it?

a : They use pre-trained network(vgg19) to extract features from content images and style images
b : Generate a random image(white noise) and try to minimize the loss between random image, content images and style images. Total loss = content_loss(content, random_image) + style_loss(style_image + random_image). style_loss could be a summation of gram matrix of many conv output, content_loss is mean square error.

3 : What kind of results are they getting?

The results is the combination of source image and the style image, the results of the paper seem rosy. However, I find out not every images work out of the box, you may need to tune this and that to make the result looks good.

4 : What previous work are they building on?

They are using a technique which called non-photorealistic rendering. I haven’t read this paper yet, have no idea how it works.

70

For those who are interested in building their own deep learning box I created a tutorial in 2 parts to install the necessary tools for deep learning on your linux machine.
Link for part 1: How to setup your own environment for deep learning - Locally
Link for part 2: How to setup your own environment for deep learning - For remote access

71

Hi @Matthew, I am hitting my head against a brick wall as I am getting the same “All Orange” images using the default notebook for neural style. Initially i thought it was a problem with keras, tensorflow or cuda causing this, because i am doing this 6 months behind the course timeline, so i went and upgraded everything, but i am still getting this problem. Do you have any insights on how to fix that. or what you did to solve it.
I am at a loss to understand the problem. If you need further screen shots etc i can provide.

edit: I see others had this issue. Im going hunting for the solution. If you or others can post a link to the solution please do. Thanks

edit2:
Ok looks like i solved it from above posts. Thanks all. Heres what i did. Changed the following loss function.

# loss = metrics.mse(layer, targ)         ## This caused the problem
loss = K.mean(metrics.mse(layer, targ))   ## This fixes the issue
grads = K.gradients(loss, model.input)
fn = K.function([model.input], [loss]+grads)
evaluator = Evaluator(fn, shp)
72

I’m having trouble understanding the how the content loss is calculated. In the notebook, it’s

layer_model = Model(model.input, layer)
targ = K.variable(layer_model.predict(img_arr))
loss = metrics.mse(layer, targ)

It seems like this is calculating the loss between the input image and the activations in the final convolution layer. But, if I understood correctly, it should be comparing the activations of the noisy image with the activations of the target image.

73

Nevermind, I get it now. The layer variable is the activations of the model on the noise image image.

74

I finally got it working! I used a picture of the golden gate bridge from here, and Van Goh’s starry night as the style. I ran it for 50 iterations, and the result was pretty cool.

bridge_styled__final.jpg1692×788 231 KB

5 Likes
75

In case anyone is confused about why this is, I dug into it a bit. I don’t know why this worked in Jeremy’s original notebook (perhaps it was an older version of Keras) but the mean squared error function is defined here. It performs a mean over the final axis, which in our case is the filters, but this is a marginal mean – it returns the MSE over each individual value rather than reducing the entire difference matrix to a single value. We then apply K.mean to this value without passing a specified axis; in tensorflow it falls back to this function, which, when an axis is not specified, reduces the entire dataset to a scalar.

It seems in the original notebook, metrics.mse was doing this by default. Something must have changed, but I’m not sure what it is. Hope this makes things a little more clear to anyone that was struggling with this issue (as I was).

.