I don’t have the output anymore b/c I’m running it with bs=24, but it was an error stack with CUD: out of memory at the bottom. I’ll paste it if it happens again
1 Like
this is the error I’m getting:
RuntimeError Traceback (most recent call last)
in
3 dls = TextDataLoaders.from_folder(untar_data(URLs.IMDB), valid=‘test’)
4 learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)
----> 5 learn.fine_tune(4, 1e-2)
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastai2/callback/schedule.py in fine_tune(self, epochs, base_lr, freeze_epochs, lr_mult, pct_start, div, **kwargs)
158 base_lr /= 2
159 self.unfreeze()
–> 160 self.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div, **kwargs)
161
162 # Cell
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastai2/callback/schedule.py in fit_one_cycle(self, n_epoch, lr_max, div, div_final, pct_start, wd, moms, cbs, reset_opt)
110 scheds = {‘lr’: combined_cos(pct_start, lr_max/div, lr_max, lr_max/div_final),
111 ‘mom’: combined_cos(pct_start, *(self.moms if moms is None else moms))}
–> 112 self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd)
113
114 # Cell
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastai2/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt)
189 try:
190 self.epoch=epoch; self(‘begin_epoch’)
–> 191 self._do_epoch_train()
192 self._do_epoch_validate()
193 except CancelEpochException: self(‘after_cancel_epoch’)
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastai2/learner.py in _do_epoch_train(self)
162 try:
163 self.dl = self.dls.train; self(‘begin_train’)
–> 164 self.all_batches()
165 except CancelTrainException: self(‘after_cancel_train’)
166 finally: self(‘after_train’)
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastai2/learner.py in all_batches(self)
140 def all_batches(self):
141 self.n_iter = len(self.dl)
–> 142 for o in enumerate(self.dl): self.one_batch(*o)
143
144 def one_batch(self, i, b):
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastai2/learner.py in one_batch(self, i, b)
146 try:
147 self._split(b); self(‘begin_batch’)
–> 148 self.pred = self.model(*self.xb); self(‘after_pred’)
149 if len(self.yb) == 0: return
150 self.loss = self.loss_func(self.pred, *self.yb); self(‘after_loss’)
/opt/conda/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
530 result = self._slow_forward(*input, **kwargs)
531 else:
–> 532 result = self.forward(*input, **kwargs)
533 for hook in self._forward_hooks.values():
534 hook_result = hook(self, input, result)
/opt/conda/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
98 def forward(self, input):
99 for module in self:
–> 100 input = module(input)
101 return input
102
/opt/conda/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
530 result = self._slow_forward(*input, **kwargs)
531 else:
–> 532 result = self.forward(*input, **kwargs)
533 for hook in self._forward_hooks.values():
534 hook_result = hook(self, input, result)
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastai2/text/models/core.py in forward(self, input)
79 #Note: this expects that sequence really begins on a round multiple of bptt
80 real_bs = (input[:,i] != self.pad_idx).long().sum()
—> 81 o = self.module(input[:real_bs,i: min(i+self.bptt, sl)])
82 if self.max_len is None or sl-i <= self.max_len:
83 outs.append(o)
/opt/conda/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
530 result = self._slow_forward(*input, **kwargs)
531 else:
–> 532 result = self.forward(*input, **kwargs)
533 for hook in self._forward_hooks.values():
534 hook_result = hook(self, input, result)
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastai2/text/models/awdlstm.py in forward(self, inp, from_embeds)
100 new_hidden = []
101 for l, (rnn,hid_dp) in enumerate(zip(self.rnns, self.hidden_dps)):
–> 102 output, new_h = rnn(output, self.hidden[l])
103 new_hidden.append(new_h)
104 if l != self.n_layers - 1: output = hid_dp(output)
/opt/conda/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
530 result = self._slow_forward(*input, **kwargs)
531 else:
–> 532 result = self.forward(*input, **kwargs)
533 for hook in self._forward_hooks.values():
534 hook_result = hook(self, input, result)
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastai2/text/models/awdlstm.py in forward(self, *args)
49 #To avoid the warning that comes because the weights aren’t flattened.
50 warnings.simplefilter(“ignore”)
—> 51 return self.module.forward(*args)
52
53 def reset(self):
/opt/conda/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
557 if batch_sizes is None:
558 result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
–> 559 self.dropout, self.training, self.bidirectional, self.batch_first)
560 else:
561 result = _VF.lstm(input, batch_sizes, hx, self._flat_weights, self.bias,
RuntimeError: CUDA out of memory. Tried to allocate 66.00 MiB (GPU 0; 7.94 GiB total capacity; 7.30 GiB already allocated; 10.38 MiB free; 7.53 GiB reserved in total by PyTorch)
That image was missing from the github repository at the begging, you can pull from the master or recopy again.
Is there a promotional credit for Paperspace?
They used to have referrals which gave a new user $10 dollar or so when using the referral link. There are a lot of these here in this thread
Just uploaded an image to test something using widgets.fileUpload but can’t find where it put it. When you upload something, where does it go?
2 Likes
Hi All,
I am getting the following error. I used the latest container mentioned above.
IN:
from fastai2.text.all import *
dls = TextDataLoaders.from_folder(untar_data(URLs.IMDB), valid=‘test’)
learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)
learn.fine_tune(4, 1e-2)
OUT:
FileNotFoundError Traceback (most recent call last)
in
1 from fastai2.text.all import *
2
----> 3 dls = TextDataLoaders.from_folder(untar_data(URLs.IMDB), valid=‘test’)
4 learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)
5 learn.fine_tune(4, 1e-2)
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastai2/text/data.py in from_folder(cls, path, train, valid, valid_pct, seed, vocab, text_vocab, is_lm, tok_tfm, seq_len, **kwargs)
207 “Create from imagenet style dataset in path with train,valid,test subfolders (or provide valid_pct).”
208 splitter = GrandparentSplitter(train_name=train, valid_name=valid) if valid_pct is None else RandomSplitter(valid_pct, seed=seed)
–> 209 blocks = [TextBlock.from_folder(path, text_vocab, is_lm, seq_len) if tok_tfm is None else TextBlock(tok_tfm, text_vocab, is_lm, seq_len)]
210 if not is_lm: blocks.append(CategoryBlock(vocab=vocab))
211 get_items = partial(get_text_files, folders=[train,valid]) if valid_pct is None else get_text_files
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastai2/text/data.py in from_folder(cls, path, vocab, is_lm, seq_len, min_freq, max_vocab, **kwargs)
196 @delegates(Tokenizer.from_folder, keep=True)
197 def from_folder(cls, path, vocab=None, is_lm=False, seq_len=72, min_freq=3, max_vocab=60000, **kwargs):
–> 198 return cls(Tokenizer.from_folder(path, **kwargs), vocab=vocab, is_lm=is_lm, seq_len=seq_len,
199 min_freq=min_freq, max_vocab=max_vocab)
200
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastai2/text/core.py in from_folder(cls, path, tok_func, rules, **kwargs)
270 output_dir = Path(ifnone(kwargs.get(‘output_dir’), path.parent/f’{path.name}_tok’))
271 if not output_dir.exists(): tokenize_folder(path, rules=rules, **kwargs)
–> 272 res = cls(get_tokenizer(tok_func, **kwargs), counter=(output_dir/fn_counter_pkl).load(),
273 lengths=(output_dir/fn_lengths_pkl).load(), rules=rules, mode=‘folder’)
274 res.path,res.output_dir = path,output_dir
/opt/conda/envs/fastai/lib/python3.7/site-packages/fastcore/utils.py in load(fn)
442 def load(fn:Path):
443 “Load a pickle file from a file name or opened file”
–> 444 if not isinstance(fn, io.IOBase): fn = open(fn,‘rb’)
445 try: return pickle.load(fn)
446 finally: fn.close()
FileNotFoundError: [Errno 2] No such file or directory: ‘/storage/data/imdb_tok/counter.pkl’
Hello,
I’m using the free paperspace gradient instance. Working well (few minor glitches with missing files, but it’s amazing to have the whole thing working in less than a minute!).
I have a question about switching notebooks:
I opened a notebook with, say, lesson 01 of the course. I used it and want to move to lesson 02. I tried to “stop and halt” the notebook but I was still “stuck” in the notebook and couldn’t go back to the directory tree screen where I can choose a different notebook from the course materials folder. I finally managed to do it by going back to the console and pressing start notebook again. Is there an easier way to switch notebooks?
Thanks for the great platform and support!
Yonatan.
1 Like
UPDATE:
after being happy with the platform, and annoyed with the switching notebooks, I decided to purchase a subscription.
Since then I have some problems.
-
I guess I don’t get the idea of the storage and notebooks. I started a notebook with a cheap CPU computer. I saved the notebook file in the storage folder and later tried to access it from a GPU instance - nothing… the file is not there. Is the storage shared between instances, or is it specific to a “notebook” (which might be a misleading name as its actually a Jupiter server, isn’t it? so I’ll call the paper space notebook NOTEBOOK from now on).
UPDATE: I discovered that some instances are in a different location than others, so they don’t share the storage. Storage sharing is crucial for work continuity so I think this should be more clear to the user when he chooses instance types, which share the storage. -
I reproduced my whole work on a GPU notebook instance (the cheap version of K80). When running, the kernel crashed. After it had crashed I could do nothing. I couldn’t restart it, I couldn’t reload or save the notebook. Every time I got back to the console and pressed “open” it showed the top bar but never loaded the notebook. I had to stop the NOTEBOOK and restart it again, which takes around 10 minutes.
-
During work some functionality suddenly disappears, such as using shift+tab to explore functions. This is one of the most useful features of Jupiter and I can’t see myself working without it.
Would appreciate your support, thanks!
1 Like
@Jess Gradient includes a few different free GPU instances. There are other more powerful instances that are not free but we are offering a promo code to the Fast.ai community: FASTAI2020
5 Likes
@Jess @miwojc We still offer referral credit where you can send credit and earn $15 per account 
You can find your referral code in the console on the settings page:
4 Likes
@yonatan365 Good question, I wish this was more obvious. Just click the Jupyter icon to go to the root of the notebook:
2 Likes
Here are some hints for Paperspace total beginners. As one, it took me a while to figure out this basic orientation.
-
Paperspace Notebooks are not the same as Jupyter Notebooks. A Paperspace notebook is a virtual environment. With a free account you can create only one. However, you can create and use multiple Jupyter notebooks within one Paperspace Notebook.
-
To work with several Jupyter notebooks at once, Ctrl-click the notebook name in the Jupyter file browser. It will open in a new tab.
-
To use the fastai book…
Open a new Terminal from the Jupyter file browser.
Navigate to the desired location for fastbook. Next…
git clone https://github.com/fastai/fastbook.git
Now you can open the fastbook chapters using the Jupyter notebook browser.
To keep the book up-to-date,
cd fastbook (wherever you placed it)
git pull
You will need to regularly update fastai2 and the course as described in the top post FAQ.
- When you get lost, a click in the upper left box that says “Gradient” will go back to the Gradient Notebook console.
- Click on “Jupyter” at top to get back to the Jupyter notebook browser.
13 Likes
Hi, yes, I was able to select the GPU and paperspace notebook. I noticed that my GPU wasn’t the P5000 which was recommended, so I just ended up creating a new paperspace notebook using the P5000 GPU and now the jupyter notebook is working and there are no memory errors. thanks for your help
1 Like
Thanks for posting the tips on Paperspace. They were really helpful to me and cleared up some confusion I had.
1 Like
@dkobran hey Daniel I am trying to run line 15 from the 01__intro notebook.
dls = TextDataLoaders.from_folder(untar_data(URLs.IMDB), valid='test')
learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)
learn.fine_tune(4, 1e-2)```
I have not encountered an error yet (meaning any CUDA out of memory message). However, the cell has been running for almost 10 min is this behavior expected? I am using the free instances and the P5000@mario_carrillo text learners take a very long time to train. Each epoch is usually around 7-10 minutes + depending on the GPU. So that bit of code right there would take 30+ minutes to run
3 Likes
In terminal, do we need to use conda fastai environment?
by typing:
bash
conda activate fastai