RNN training error - Lesson-6

Part 1 (2017)
1

Hi Everyone,

I ma facing an error that I couldn’t fix myself neither got anything from stackoverflow or this forum. I wanted to see if anyone else faced the same error or can help me solving this.

I am following lesson code without any changes and still getting this error:

Code:

n_hidden, n_fac, cs, vocab_size = (256, 42, 8, 86)

model=Sequential([
Embedding(vocab_size, n_fac, input_length=cs),
SimpleRNN(n_hidden, activation=‘relu’, inner_init=‘identity’),
Dense(vocab_size, activation=‘softmax’)
])

model.compile(loss = ‘sparse_categorical_crossentropy’, optimizer=Adam())
model.fit(np.stack(xs,1), y, batch_size=64, nb_epoch=8)

Error:

poch 1/8

IndexError Traceback (most recent call last)
in ()
----> 1 model.fit(np.stack(xs,1), y, batch_size=64, nb_epoch=8)

/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/keras/models.pyc in fit(self, x, y, batch_size, nb_epoch, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, **kwargs)
650 shuffle=shuffle,
651 class_weight=class_weight,
–> 652 sample_weight=sample_weight)
653
654 def evaluate(self, x, y, batch_size=32, verbose=1,

/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in fit(self, x, y, batch_size, nb_epoch, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch)
1109 val_f=val_f, val_ins=val_ins, shuffle=shuffle,
1110 callback_metrics=callback_metrics,
-> 1111 initial_epoch=initial_epoch)
1112
1113 def evaluate(self, x, y, batch_size=32, verbose=1, sample_weight=None):

/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in _fit_loop(self, f, ins, out_labels, batch_size, nb_epoch, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch)
824 batch_logs[‘size’] = len(batch_ids)
825 callbacks.on_batch_begin(batch_index, batch_logs)
–> 826 outs = f(ins_batch)
827 if type(outs) != list:
828 outs = [outs]

/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.pyc in call(self, inputs)
809 def call(self, inputs):
810 assert type(inputs) in {list, tuple}
–> 811 return self.function(*inputs)
812
813

/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.pyc in call(self, *args, **kwargs)
869 node=self.fn.nodes[self.fn.position_of_error],
870 thunk=thunk,
–> 871 storage_map=getattr(self.fn, ‘storage_map’, None))
872 else:
873 # old-style linkers raise their own exceptions

/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/theano/gof/link.pyc in raise_with_op(node, thunk, exc_info, storage_map)
312 # extra long error message in that case.
313 pass
–> 314 reraise(exc_type, exc_value, exc_trace)
315
316

/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.pyc in call(self, *args, **kwargs)
857 t0_fn = time.time()
858 try:
–> 859 outputs = self.fn()
860 except Exception:
861 if hasattr(self.fn, ‘position_of_error’):

/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/theano/gof/op.pyc in rval(p, i, o, n)
910 # default arguments are stored in the closure of rval
911 def rval(p=p, i=node_input_storage, o=node_output_storage, n=node):
–> 912 r = p(n, [x[0] for x in i], o)
913 for o in node.outputs:
914 compute_map[o][0] = True

/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/theano/tensor/subtensor.pyc in perform(self, node, inputs, out_)
2284
2285 if self.set_instead_of_inc:
-> 2286 out[0][inputs[2:]] = inputs[1]
2287 elif inplace_increment is not None:
2288 inplace_increment(out[0], tuple(inputs[2:]), inputs[1])

IndexError: index 94 is out of bounds for axis 1 with size 86
Apply node that caused the error: AdvancedIncSubtensor{inplace=False, set_instead_of_inc=True}(Alloc.0, TensorConstant{1}, ARange{dtype=‘int64’}.0, Elemwise{Cast{int32}}.0)
Toposort index: 77
Inputs types: [TensorType(float32, matrix), TensorType(int8, scalar), TensorType(int64, vector), TensorType(int32, vector)]
Inputs shapes: [(64, 86), (), (64,), (64,)]
Inputs strides: [(344, 4), (), (8,), (4,)]
Inputs values: [‘not shown’, array(1, dtype=int8), ‘not shown’, ‘not shown’]
Outputs clients: [[Reshape{2}(AdvancedIncSubtensor{inplace=False, set_instead_of_inc=True}.0, MakeVector{dtype=‘int64’}.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
File “/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py”, line 2827, in run_ast_nodes
if self.run_code(code, result):
File “/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py”, line 2881, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File “”, line 1, in
model.compile(loss = ‘sparse_categorical_crossentropy’, optimizer=Adam())
File “/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/keras/models.py”, line 578, in compile
**kwargs)
File “/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/keras/engine/training.py”, line 604, in compile
sample_weight, mask)
File “/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/keras/engine/training.py”, line 303, in weighted
score_array = fn(y_true, y_pred)
File “/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/keras/objectives.py”, line 45, in sparse_categorical_crossentropy
return K.sparse_categorical_crossentropy(y_pred, y_true)
File “/Users/trinakarmakar/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py”, line 1079, in sparse_categorical_crossentropy
target = T.extra_ops.to_one_hot(target, nb_class=output.shape[-1])

HINT: Use the Theano flag ‘exception_verbosity=high’ for a debugprint and storage map footprint of this apply node.

2

That line is the key part of your error. I haven’t used the Embedding Layer before, does it automatically strip out words that aren’t in its vocabulary? You may need to make sure that your input consists only of integers [0,86) - set aside a token for out of vocabulary words.

I would also check the shape of your input against mode.summary().

3

Thanks, I have diffarent text file to working with which I myself forgot at the end. BTW, I could not access S3 text file used in the session-6 course at the first place, so I end up creating my own Nietzsche text file to work on.
Now, I changed vocab_size to 94 and rerunning my training process–finger crossed. Thanks for your reply.

4

Hah!! hit the brick again.

Layer (type) Output Shape Param # Connected to

embedding_13 (Embedding) (None, 8, 42) 4536 embedding_input_2[0][0]

simplernn_2 (SimpleRNN) (None, 256) 76544 embedding_13[0][0]

dense_8 (Dense) (None, 108) 27756 simplernn_2[0][0]

Total params: 108836

np.stack(xs,1).shape
(66216, 8)
y.shape
(66216,)

ndexError: index 108 is out of bounds for size 108

5

@Gkarmakar Did you ever get a solution to this problem? I’m running into the same problem. I’m not using RNN yet. I’m simply hand creating the model for the third item to be predicted given first 2, just like @jeremy taught in lesson 6 . I’m using my own data. Here’s what I have:

n_fac = 100
def embedding_input(name, n_in, n_out):
inp = Input(shape=(1,), dtype=‘int64’, name=name)
emb = Embedding(n_in, n_out, input_length=1)(inp)
return inp, Flatten()(emb)

vocab_size = len(uniqueArtifacts)
x1_in, x1 = embedding_input(‘x1’, vocab_size, n_fac)
x2_in, x2 = embedding_input(‘x2’, vocab_size, n_fac)
n_hidden = 512
dense_in = Dense(n_hidden, activation=‘relu’)
x1_hidden = dense_in(x1)
dense_hidden = Dense(n_hidden, activation=‘tanh’)
x2_dense = dense_in(x2)
hidden_2 = dense_hidden(x1_hidden)
x2_hidden = merge([x2_dense, hidden_2])
dense_out = Dense(vocab_size, activation=‘softmax’)
x3_out = dense_out(x2_hidden)

model = Model([x1_in, x2_in], x3_out)
model.compile(loss=‘sparse_categorical_crossentropy’, optimizer=Adam())
model.fit([x1_dat, x2_dat], y_dat, batch_size=10, nb_epoch=1, verbose = 1)

Here’s the error I get:

IndexError Traceback (most recent call last)
in ()
----> 1 model.fit([x1_dat, x2_dat], y_dat, batch_size=10, nb_epoch=1, verbose = 1)

/home/prateek/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in fit(self, x, y, batch_size, nb_epoch, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch)
1194 val_f=val_f, val_ins=val_ins, shuffle=shuffle,
1195 callback_metrics=callback_metrics,
-> 1196 initial_epoch=initial_epoch)
1197
1198 def evaluate(self, x, y, batch_size=32, verbose=1, sample_weight=None):

/home/prateek/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in _fit_loop(self, f, ins, out_labels, batch_size, nb_epoch, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch)
889 batch_logs[‘size’] = len(batch_ids)
890 callbacks.on_batch_begin(batch_index, batch_logs)
–> 891 outs = f(ins_batch)
892 if not isinstance(outs, list):
893 outs = [outs]

/home/prateek/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.pyc in call(self, inputs)
957 def call(self, inputs):
958 assert isinstance(inputs, (list, tuple))
–> 959 return self.function(*inputs)
960
961

/home/prateek/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.pyc in call(self, *args, **kwargs)
896 node=self.fn.nodes[self.fn.position_of_error],
897 thunk=thunk,
–> 898 storage_map=getattr(self.fn, ‘storage_map’, None))
899 else:
900 # old-style linkers raise their own exceptions

/home/prateek/anaconda2/lib/python2.7/site-packages/theano/gof/link.pyc in raise_with_op(node, thunk, exc_info, storage_map)
323 # extra long error message in that case.
324 pass
–> 325 reraise(exc_type, exc_value, exc_trace)
326
327

/home/prateek/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.pyc in call(self, *args, **kwargs)
882 try:
883 outputs =
–> 884 self.fn() if output_subset is None else
885 self.fn(output_subset=output_subset)
886 except Exception:

/home/prateek/anaconda2/lib/python2.7/site-packages/theano/gof/op.pyc in rval(p, i, o, n)
870 # default arguments are stored in the closure of rval
871 def rval(p=p, i=node_input_storage, o=node_output_storage, n=node):
–> 872 r = p(n, [x[0] for x in i], o)
873 for o in node.outputs:
874 compute_map[o][0] = True

/home/prateek/anaconda2/lib/python2.7/site-packages/theano/tensor/subtensor.pyc in perform(self, node, inputs, out_)
2240
2241 if self.set_instead_of_inc:
-> 2242 out[0][inputs[2:]] = inputs[1]
2243 elif config.cxx:
2244 inplace_increment(out[0], tuple(inputs[2:]), inputs[1])

IndexError: index 1864798 is out of bounds for axis 1 with size 2392
Apply node that caused the error: AdvancedIncSubtensor{inplace=False, set_instead_of_inc=True}(Alloc.0, TensorConstant{1}, ARange{dtype=‘int64’}.0, Elemwise{Cast{int32}}.0)
Toposort index: 69
Inputs types: [TensorType(float32, matrix), TensorType(int8, scalar), TensorType(int64, vector), TensorType(int32, vector)]
Inputs shapes: [(10, 2392), (), (10,), (10,)]
Inputs strides: [(9568, 4), (), (8,), (4,)]
Inputs values: [‘not shown’, array(1, dtype=int8), ‘not shown’, ‘not shown’]
Outputs clients: [[GpuFromHost(AdvancedIncSubtensor{inplace=False, set_instead_of_inc=True}.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
File “/home/prateek/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py”, line 2717, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File “/home/prateek/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py”, line 2827, in run_ast_nodes
if self.run_code(code, result):
File “/home/prateek/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py”, line 2881, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File “”, line 1, in
model.compile(loss=‘sparse_categorical_crossentropy’, optimizer=Adam())
File “/home/prateek/anaconda2/lib/python2.7/site-packages/keras/engine/training.py”, line 667, in compile
sample_weight, mask)
File “/home/prateek/anaconda2/lib/python2.7/site-packages/keras/engine/training.py”, line 318, in weighted
score_array = fn(y_true, y_pred)
File “/home/prateek/anaconda2/lib/python2.7/site-packages/keras/objectives.py”, line 41, in sparse_categorical_crossentropy
return K.sparse_categorical_crossentropy(y_pred, y_true)
File “/home/prateek/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py”, line 1249, in sparse_categorical_crossentropy
target = T.extra_ops.to_one_hot(target, nb_class=output.shape[-1])

HINT: Use the Theano flag ‘exception_verbosity=high’ for a debugprint and storage map footprint of this apply node.