From what I remember, the original Starry Night image was pretty big. Jeremy divided by some number (3.5) that brought it down in size to something closer to the content image. I’m guessing the Simpson style image was also pretty big, but not quite as big, and so it was only divided by 2.7. A heuristic is probably to get the style image close to the resolution of your content image (although, it might be a fun experiment to use style images with very different resolutions than the content image).
Hi @resdntalien, I think the random image that we start with has to be of same size as content image.Because content loss requires both of them to be same shape.But in case of style loss, the random image need not have the same size as style image as long as they are considered from the same layer.This is because we are using gram matrix which will have dimensions of num_filtersxnum_filters
Hi Jeremy, I’m going through the MOOC right now and I’m getting a similar issue but when I remove the /100 it still doesn’t work. My code is here: https://github.com/zaoyang/fast_ai_course/tree/master/part2_orig
https://github.com/zaoyang/fast_ai_course/blob/master/part2_orig/neural-style.ipynb
The only thing I have different is the requirements with Keras and I did some minor tweaks to make it work with the latest Keras. Any idea what’s wrong?
In requirements.txt, it has my environment. I’m tried this locally on Mac and on the AWS server with Ubuntu.
Conda environment:
PIP environment:
Both get this yellow image issues. Is this an environment issue or code issue?
@jeremy The other issues I’m running into is the following. This is happening on both Ubuntu and Mac as well. I didn’t change anything. So I’m not sure if this is an issue with changing from Theano to TF or another issues. Online it says that it’s bad initalizations of shapes but not sure what the issue could be.
indent preformatted text by 4 spaces
---------------------------------------------------------------------------
InvalidArgumentError Traceback (most recent call last)
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1038 try:
-> 1039 return fn(*args)
1040 except errors.OpError as e:
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
1020 feed_dict, fetch_list, target_list,
-> 1021 status, run_metadata)
1022
~/anaconda3/envs/ml_py3/lib/python3.6/contextlib.py in __exit__(self, type, value, traceback)
88 try:
---> 89 next(self.gen)
90 except StopIteration:
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py in raise_exception_on_not_ok_status()
465 compat.as_text(pywrap_tensorflow.TF_Message(status)),
--> 466 pywrap_tensorflow.TF_GetCode(status))
467 finally:
InvalidArgumentError: Incompatible shapes: [64] vs. [128]
[[Node: gradients_1/add_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _class=["loc:@add_1"], _device="/job:localhost/replica:0/task:0/cpu:0"](gradients_1/add_1_grad/Shape, gradients_1/add_1_grad/Shape_1)]]
During handling of the above exception, another exception occurred:
InvalidArgumentError Traceback (most recent call last)
<ipython-input-71-d733ab4b440a> in <module>()
----> 1 x = solve_image(evaluator, iterations, x)
<ipython-input-27-99c5991ad031> in solve_image(eval_obj, niter, x)
2 for i in range(niter):
3 x, min_val, info = fmin_l_bfgs_b(eval_obj.loss, x.flatten(),
----> 4 fprime=eval_obj.grads, maxfun=20)
5 x = np.clip(x, -127,127)
6 print('Current loss value:', min_val)
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py in fmin_l_bfgs_b(func, x0, fprime, args, approx_grad, bounds, m, factr, pgtol, epsilon, iprint, maxfun, maxiter, disp, callback, maxls)
191
192 res = _minimize_lbfgsb(fun, x0, args=args, jac=jac, bounds=bounds,
--> 193 **opts)
194 d = {'grad': res['jac'],
195 'task': res['message'],
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py in _minimize_lbfgsb(fun, x0, args, jac, bounds, disp, maxcor, ftol, gtol, eps, maxfun, maxiter, iprint, callback, maxls, **unknown_options)
326 # until the completion of the current minimization iteration.
327 # Overwrite f and g:
--> 328 f, g = func_and_grad(x)
329 elif task_str.startswith(b'NEW_X'):
330 # new iteration
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py in func_and_grad(x)
276 else:
277 def func_and_grad(x):
--> 278 f = fun(x, *args)
279 g = jac(x, *args)
280 return f, g
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/scipy/optimize/optimize.py in function_wrapper(*wrapper_args)
290 def function_wrapper(*wrapper_args):
291 ncalls[0] += 1
--> 292 return function(*(wrapper_args + args))
293
294 return ncalls, function_wrapper
<ipython-input-25-95afa49bd58f> in loss(self, x)
3
4 def loss(self, x):
----> 5 loss_, self.grad_values = self.f([x.reshape(self.shp)])
6 return loss_.astype(np.float64)
7
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py in __call__(self, inputs)
2227 session = get_session()
2228 updated = session.run(self.outputs + [self.updates_op],
-> 2229 feed_dict=feed_dict)
2230 return updated[:len(self.outputs)]
2231
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
776 try:
777 result = self._run(None, fetches, feed_dict, options_ptr,
--> 778 run_metadata_ptr)
779 if run_metadata:
780 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
980 if final_fetches or final_targets:
981 results = self._do_run(handle, final_targets, final_fetches,
--> 982 feed_dict_string, options, run_metadata)
983 else:
984 results = []
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1030 if handle is None:
1031 return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1032 target_list, options, run_metadata)
1033 else:
1034 return self._do_call(_prun_fn, self._session, handle, feed_dict,
~/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1050 except KeyError:
1051 pass
-> 1052 raise type(e)(node_def, op, message)
1053
1054 def _extend_graph(self):
InvalidArgumentError: Incompatible shapes: [64] vs. [128]
[[Node: gradients_1/add_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _class=["loc:@add_1"], _device="/job:localhost/replica:0/task:0/cpu:0"](gradients_1/add_1_grad/Shape, gradients_1/add_1_grad/Shape_1)]]
Caused by op 'gradients_1/add_1_grad/BroadcastGradientArgs', defined at:
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in <module>
app.launch_new_instance()
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 477, in start
ioloop.IOLoop.instance().start()
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/zmq/eventloop/ioloop.py", line 177, in start
super(ZMQIOLoop, self).start()
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tornado/ioloop.py", line 888, in start
handler_func(fd_obj, events)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell
handler(stream, idents, msg)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 196, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2698, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2802, in run_ast_nodes
if self.run_code(code, result):
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2862, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-67-de37f8a93ea6>", line 2, in <module>
grads = K.gradients(loss, model.input)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2264, in gradients
return tf.gradients(loss, variables, colocate_gradients_with_ops=True)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 560, in gradients
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 368, in _MaybeCompile
return grad_fn() # Exit early
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 560, in <lambda>
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py", line 598, in _AddGrad
rx, ry = gen_array_ops._broadcast_gradient_args(sx, sy)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 411, in _broadcast_gradient_args
name=name)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
op_def=op_def)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
self._traceback = _extract_stack()
...which was originally created as op 'add_1', defined at:
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
[elided 18 identical lines from previous traceback]
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2862, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-67-de37f8a93ea6>", line 1, in <module>
loss = sum(style_loss(l1[0], l2[0]) for l1,l2 in zip(layers, targs))
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 821, in binary_op_wrapper
return func(x, y, name=name)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 73, in add
result = _op_def_lib.apply_op("Add", x=x, y=y, name=name)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
op_def=op_def)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/Users/zaoyang/anaconda3/envs/ml_py3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): Incompatible shapes: [64] vs. [128]
[[Node: gradients_1/add_1_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _class=["loc:@add_1"], _device="/job:localhost/replica:0/task:0/cpu:0"](gradients_1/add_1_grad/Shape, gradients_1/add_1_grad/Shape_1)]]
Hi @zaoyang, In Lesson 8 discussions some of us including myself had same problems.Check lesson 8 discussions for fix
thanks I’ll check it out. Did the Keras 2 implementation from Romano work for you? It wasn’t exactly the same issue. It’s the orange blob.
Hi @zaoyang, try , loss = K.mean(metrics.mse(layer, targ)) .I think the current content loss function you have doesn’t return a single value
Here:
Here’s my conda and pip environments:
Thank you that worked. That removed the orange output. I just don’t understand why that worked.
Hi @zaoyang, The loss returned by the original function won’t be a single value if you are using keras 2. So I had to wrap a K.mean around it
k Thank you appreciate it. I have the invalid argument as well. Not sure if encountered it. I’ll try romano’s code unless you have a quick suggestion.
My answers of question 3, anyone who find something wrong or incomplete, please help me correct them, thanks.
1 : What problem are they solving?
They figure out an algorithm which could generate artistic image by deep learning
2 : What is the general idea that they are using to solve it?
a : They use pre-trained network(vgg19) to extract features from content images and style images
b : Generate a random image(white noise) and try to minimize the loss between random image, content images and style images. Total loss = content_loss(content, random_image) + style_loss(style_image + random_image). style_loss could be a summation of gram matrix of many conv output, content_loss is mean square error.
3 : What kind of results are they getting?
The results is the combination of source image and the style image, the results of the paper seem rosy. However, I find out not every images work out of the box, you may need to tune this and that to make the result looks good.
4 : What previous work are they building on?
They are using a technique which called non-photorealistic rendering. I haven’t read this paper yet, have no idea how it works.
For those who are interested in building their own deep learning box I created a tutorial in 2 parts to install the necessary tools for deep learning on your linux machine.
Link for part 1: How to setup your own environment for deep learning - Locally
Link for part 2: How to setup your own environment for deep learning - For remote access
Hi @Matthew, I am hitting my head against a brick wall as I am getting the same “All Orange” images using the default notebook for neural style. Initially i thought it was a problem with keras, tensorflow or cuda causing this, because i am doing this 6 months behind the course timeline, so i went and upgraded everything, but i am still getting this problem. Do you have any insights on how to fix that. or what you did to solve it.
I am at a loss to understand the problem. If you need further screen shots etc i can provide.
edit: I see others had this issue. Im going hunting for the solution. If you or others can post a link to the solution please do. Thanks
edit2:
Ok looks like i solved it from above posts. Thanks all. Heres what i did. Changed the following loss function.
# loss = metrics.mse(layer, targ) ## This caused the problem
loss = K.mean(metrics.mse(layer, targ)) ## This fixes the issue
grads = K.gradients(loss, model.input)
fn = K.function([model.input], [loss]+grads)
evaluator = Evaluator(fn, shp)
I’m having trouble understanding the how the content loss is calculated. In the notebook, it’s
layer_model = Model(model.input, layer)
targ = K.variable(layer_model.predict(img_arr))
loss = metrics.mse(layer, targ)
It seems like this is calculating the loss between the input image and the activations in the final convolution layer. But, if I understood correctly, it should be comparing the activations of the noisy image with the activations of the target image.
Nevermind, I get it now. The layer
variable is the activations of the model on the noise image image.
I finally got it working! I used a picture of the golden gate bridge from here, and Van Goh’s starry night as the style. I ran it for 50 iterations, and the result was pretty cool.
In case anyone is confused about why this is, I dug into it a bit. I don’t know why this worked in Jeremy’s original notebook (perhaps it was an older version of Keras) but the mean squared error function is defined here. It performs a mean over the final axis, which in our case is the filters, but this is a marginal mean – it returns the MSE over each individual value rather than reducing the entire difference matrix to a single value. We then apply K.mean
to this value without passing a specified axis; in tensorflow it falls back to this function, which, when an axis is not specified, reduces the entire dataset to a scalar.
It seems in the original notebook, metrics.mse
was doing this by default. Something must have changed, but I’m not sure what it is. Hope this makes things a little more clear to anyone that was struggling with this issue (as I was).
.
Anyone else having trouble reaching the level of performance displayed in Jeremy’s notebook, even when running the code (nearly*) unchanged? My best losses are in the hundreds – generally 400-500 – while Jeremy’s gets down to around 5. I must be missing something, or a default in one of the underlying libraries may have changed and needs to be adjusted. Just wondering if anyone else has as a good solution.
*the only changes I’ve made to Jeremy’s code are those meant to switch to Keras 2 (e.g., wrapping all the metrics.mse
calls in a K.mean
). It’s possible that I messed one of these up and that’s causing the issue, but I haven’t been able to spot it.
I am working hard to make the vgg16 to work with custom dataset but there are some compatibility issue that I am facing right now , can anyone suggest the requirements.txt that is needed to run the vgg16 model for my custom dataset.
I was trying to edit the vgg16.py file to make it work with my dataset but value error as well as InvalidArgument error is poping up which is making me nuts while debugging. please do specify the packages that are needed