I’m getting run time errors when running lesson 1 code. Any suggestions on how to resolve this?
I have the same question! I already asked it in YouTube comments as well as in this forum in another topic. If I get an answer over there I’ll let you know.
Are you running this at your local machine or on AWS? Try to set the batch size to a smaller number if you do not have a GPU with lots of memory.
I didn’t see any css. what part?
I’m running it on my local machine. I have NVIDIA GTX965 4GB . I had changed the batch size from 64 to 32 to 16 but it makes no difference.
I would have thought that my GPU could handle this relatively undemanding task.
Reducing the batch size actually worked. I just forgot to re run the preceding code segments. I had the batch size set to 16 at that point and it took about 8 minutes to process. I will try with a larger batch sizes of 32.
Thanks for the tip, Maxim.
1 Like
@dhatawesomedude what are you talking about? Issue is resolved though. By uploading the downloaded file.
@jeremy I have worked through lesson one successfully. My jupyter notebook will effectively classify the cats vs. dogs data set with 97% accuracy as described in the video. I am trying to run the Vgg16 model on a data set of my own. I structured the directories to mimic that of the dogscats directory. The images show up as expected under the plots(imgs, titles=labels) line. However, the predicted labels are not the names of my two directories. Do I understand correctly that Vgg16 is pre-trained to recognize only images contained on ImageNet? The images I am trying to use are not from ImageNet.
If that is the case, do you have suggestions or resources on how to train Vgg16 (or something similar) for a set of images not contained on ImageNet.
I am running aws p2 and am using an AWS server.
Pop the last dense layer and add one with the number of categories in your set. Most of the course is about fine tuning and transfer learning, which is what you are looking for.
to do predictions with your own labels, you need to do 2 things. firstly, put your test data in directory under /test/unknown/, where /test is in the same level as /train, /valid, /sample. secondly, you need to set test_path, and pass that into a vgg.test function.
batches, preds = vgg.test(test_path, batch_size=batch_size*2)
See “Generate Predictions” section in notebook dogs_cats_redux.ipynb
1 Like
I’m getting this error, which no one else in this thread has gotten ^_^;; so I must be in the wrong. However I’m running the notebook as-is. I setup tensorflow using docker, got all the libs installed, etc. The error:
“ValueError: Negative dimension size caused by subtracting 2 from 1 for ‘MaxPool_19’ (op: ‘MaxPool’) with input shapes: [?,1,112,128].”
ValueErrorTraceback (most recent call last)
<ipython-input-57-2b6861506a11> in <module>()
----> 1 vgg = Vgg16()
2 # Grab a few images at a time for training and validation.
3 # NB: They must be in subdirectories named based on their category
4 batches = vgg.get_batches(path+'train', batch_size=batch_size)
5 val_batches = vgg.get_batches(path+'valid', batch_size=batch_size*2)
/notebooks/courses/deeplearning1/nbs/vgg16.py in __init__(self)
31 def __init__(self):
32 self.FILE_PATH = 'http://www.platform.ai/models/'
---> 33 self.create()
34 self.get_classes()
35
/notebooks/courses/deeplearning1/nbs/vgg16.py in create(self)
69
70 self.ConvBlock(2, 64)
---> 71 self.ConvBlock(2, 128)
72 self.ConvBlock(3, 256)
73 self.ConvBlock(3, 512)
/notebooks/courses/deeplearning1/nbs/vgg16.py in ConvBlock(self, layers, filters)
55 model.add(ZeroPadding2D((1, 1)))
56 model.add(Convolution2D(filters, 3, 3, activation='relu'))
---> 57 model.add(MaxPooling2D((2, 2), strides=(2, 2)))
58
59
/root/anaconda2/lib/python2.7/site-packages/keras/models.pyc in add(self, layer)
330 output_shapes=[self.outputs[0]._keras_shape])
331 else:
--> 332 output_tensor = layer(self.outputs[0])
333 if isinstance(output_tensor, list):
334 raise TypeError('All layers in a Sequential model '
/root/anaconda2/lib/python2.7/site-packages/keras/engine/topology.pyc in __call__(self, x, mask)
570 if inbound_layers:
571 # This will call layer.build() if necessary.
--> 572 self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
573 # Outputs were already computed when calling self.add_inbound_node.
574 outputs = self.inbound_nodes[-1].output_tensors
/root/anaconda2/lib/python2.7/site-packages/keras/engine/topology.pyc in add_inbound_node(self, inbound_layers, node_indices, tensor_indices)
633 # creating the node automatically updates self.inbound_nodes
634 # as well as outbound_nodes on inbound layers.
--> 635 Node.create_node(self, inbound_layers, node_indices, tensor_indices)
636
637 def get_output_shape_for(self, input_shape):
/root/anaconda2/lib/python2.7/site-packages/keras/engine/topology.pyc in create_node(cls, outbound_layer, inbound_layers, node_indices, tensor_indices)
164
165 if len(input_tensors) == 1:
--> 166 output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
167 output_masks = to_list(outbound_layer.compute_mask(input_tensors[0], input_masks[0]))
168 # TODO: try to auto-infer shape
/root/anaconda2/lib/python2.7/site-packages/keras/layers/pooling.pyc in call(self, x, mask)
158 strides=self.strides,
159 border_mode=self.border_mode,
--> 160 dim_ordering=self.dim_ordering)
161 return output
162
/root/anaconda2/lib/python2.7/site-packages/keras/layers/pooling.pyc in _pooling_function(self, inputs, pool_size, strides, border_mode, dim_ordering)
208 output = K.pool2d(inputs, pool_size, strides,
209 border_mode, dim_ordering,
--> 210 pool_mode='max')
211 return output
212
/root/anaconda2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.pyc in pool2d(x, pool_size, strides, border_mode, dim_ordering, pool_mode)
2784
2785 if pool_mode == 'max':
-> 2786 x = tf.nn.max_pool(x, pool_size, strides, padding=padding)
2787 elif pool_mode == 'avg':
2788 x = tf.nn.avg_pool(x, pool_size, strides, padding=padding)
/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.pyc in max_pool(value, ksize, strides, padding, data_format, name)
1615 padding=padding,
1616 data_format=data_format,
-> 1617 name=name)
1618
1619
/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.pyc in _max_pool(input, ksize, strides, padding, data_format, name)
1596 result = _op_def_lib.apply_op("MaxPool", input=input, ksize=ksize,
1597 strides=strides, padding=padding,
-> 1598 data_format=data_format, name=name)
1599 return result
1600
/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.pyc in apply_op(self, op_type_name, name, **keywords)
757 op = g.create_op(op_type_name, inputs, output_types, name=scope,
758 input_types=input_types, attrs=attr_protos,
--> 759 op_def=op_def)
760 if output_structure:
761 outputs = op.outputs
/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.pyc in create_op(self, op_type, inputs, dtypes, input_types, name, attrs, op_def, compute_shapes, compute_device)
2240 original_op=self._default_original_op, op_def=op_def)
2241 if compute_shapes:
-> 2242 set_shapes_for_outputs(ret)
2243 self._add_op(ret)
2244 self._record_op_seen_by_control_dependencies(ret)
/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.pyc in set_shapes_for_outputs(op)
1615 shape_func = _call_cpp_shape_fn_and_require_op
1616
-> 1617 shapes = shape_func(op)
1618 if shapes is None:
1619 raise RuntimeError(
/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.pyc in call_with_requiring(op)
1566
1567 def call_with_requiring(op):
-> 1568 return call_cpp_shape_fn(op, require_shape_fn=True)
1569
1570 _call_cpp_shape_fn_and_require_op = call_with_requiring
/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.pyc in call_cpp_shape_fn(op, input_tensors_needed, input_tensors_as_shapes_needed, debug_python_shape_fn, require_shape_fn)
608 res = _call_cpp_shape_fn_impl(op, input_tensors_needed,
609 input_tensors_as_shapes_needed,
--> 610 debug_python_shape_fn, require_shape_fn)
611 if not isinstance(res, dict):
612 # Handles the case where _call_cpp_shape_fn_impl calls unknown_shape(op).
/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.pyc in _call_cpp_shape_fn_impl(op, input_tensors_needed, input_tensors_as_shapes_needed, debug_python_shape_fn, require_shape_fn)
673 missing_shape_fn = True
674 else:
--> 675 raise ValueError(err.message)
676
677 if missing_shape_fn:
ValueError: Negative dimension size caused by subtracting 2 from 1 for 'MaxPool_19' (op: 'MaxPool') with input shapes: [?,1,112,128].
Any ideas? I’m running all code as-is, and tried decreasing batch size to 32, 16, 8, no change. Also tried with the sample dataset instead of the full one, no change, still get the same error. =\
You should use theano as a backend, not tensorflow. You are getting the error because the different image array ordering. https://keras.io/backend/
@haresenpai Please check the values in your .theanorc file. You can also scroll up and see more details about this issue.
Thanks for the hint =). I was able to get it working by changing the image_dim_ordering parameter in .keras to th (since the notebook code was written with that in mind) while keeping the backend tensoflow.
This part of the docs helped:
For 2D data (e.g. image), “tf” assumes (rows, cols, channels) while “th” assumes (channels, rows, cols).
For 3D data, “tf” assumes (conv_dim1, conv_dim2, conv_dim3, channels) while “th” assumes (channels, conv_dim1, conv_dim2, conv_dim3).
2 Likes
Ok, so for this class, I reinstalled Mint on my laptop. However, every time I turn off my laptop and try to get started again, I have to re “source aws-alias.sh”. Is there a smarter way, so I dont have to every time?
Thanks,
Dan
add ‘source {directory_name}/aws-alias.sh’ to ~/.bashrc or ~/.bash_profile
1 Like
I’ve set up the t2 and p2 instances. I’ve got the juypter notebooks running. I’m not sure if I should be downloading Cats vs. Dogs or the redux version? Also the video shows the data coming from platform which I understand doesn’t exist anymore. And/or the structure of the data has changed since then.
Where and how do I get the right data for lesson 1?