You downloaded the wrong file (I think you actually have Jeremy’s submission file for the competition from his notebook).
2 Likes
weird!: Please type in a new cell:
!cat {path/'train.csv'} | head
…wanna be sure you’re loading the right file.
1 Like
… best explanation of what self-attention is, and why its the heart of understanding the transformer.
4 Likes
To gild @wgpubs ’ lilly, another fantastic beginner resource to start understanding Transformers under the hood is this post / video by Dale Markowitz over at Google:
and
I can thoroughly recommend all of Dale’s work – she’s great at getting folks started with real-world projects in a tractable way:
12 Likes
I made one small change to Jeremy’s Kaggle NLP notebook from this:
model_nm = 'microsoft/deberta-v3-small'
To this:
model_nm = 'distilroberta-base'
The tokenizer with this model is different but was still working.
But then I got an error during the first epoch after running >> trainer.train();
This reply to a week 2 question helped correct the problem.
What other issues might occur when using different models?
Continuing on the sharing of Transformers resources, I had a whole thread of awesome resources online:
15 Likes
Yes! That’s what it was! I had the wrong train.csv file.
Thank you very much all that helped! Love you all!
I might come back for more help though, stay tuned 
5 Likes
Also, an interesting post describing transformers architecture, starting from very basic things:
https://e2eml.school/transformers.html
2 Likes
Love this Yannic’s classic paper review: it explains two of the fundamental concepts in transformer’s architecture:
- positional encoding
- key/value queries
That’s my thought: at the end of the day, positional encoding is just feature engineering on token position: same idea of breaking a date field into multiple columns (day, day_of_week, month, …).
9 Likes
I am trying to submit the notebook on kaggle but this error message comes up: Cannot submit Your Notebook cannot use internet access in this competition. Please disable internet in the Notebook editor and save a new version.
I have disconnected from the internet but some FASTAI functions require internet.
I tried this solution fastai_offline User Guide ✔ | Kaggle but still getting this error.
Help please!
It’s telling you the problem - you n eed to disable the ‘internet’ option for this notebook. It’s in the options on the top right of the main kaggle window
How do you decide whether you can get rid of outliers? When you removed those values it made your score go up, but those values will probably exist in the test set as well so do you remove the row entirely or do you use a different method for handling outliers
8 Likes
I did that but some FASTAI function requires internet… I think…
You should try out this guide instead:
Basically, you need to add the model as a dataset so you can use it offline.
10 Likes
Trying to run “Getting started with NLP” via Kaggle; I bump into an error at line tokz = AutoTokenizer.from_pretrained(model_nm) with a message of ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on. I am a bit surprised this was not already reported.
I mostly focus the live stream rather than running, so this will wait, but mentioning in case others have the same issue.
How do you go about determining and effective batchsize for GPU on whatever service we are using?
2 Likes
You don’t have internet connection in your notebook. You need to enable it, and you might need to verify your identity to do so.
If you are submitting your notebook, you technically need it to be offline and not accessing the internet, so you instead can add the model as a dataset to your notebook, something like this guide:
2 Likes