How does unseen token trained in model? I assume it is a unseen data so no gradient is ever backprop to it, so it is not updated at all. Is the weight initalized mean of embedding or just random initial weight?
Maybe similar as with batch normalization - with larger dropout you may be able to use larger learning rate and thus train faster. But this is just a guess.
Plus I might be able to determine a bias against women or a Russophobia but not as good at determining that something is biased against native americans - simply because I may not have as much exposure to what that kind of bias looks like.So it’s not just about finding testing volunteers but actively seeking out diverse testers and listening to suggestions seriously.
I’ve just updated the notebook with the link to the dataset used.
1 Like
I’m using an env for fastai on rossman_data_clean and it seems isoweek is not compatible
conda install -c auto isoweek
Solving environment: failed
UnsatisfiableError: The following specifications were found to be in conflict:
- isoweek
- pytorch-nightly -> *[track_features=cuda92]
Use "conda info <package>" to see the dependencies for each package.You can pip install isoweek, or include it in the pip section of your environment.yml if you use one.
1 Like
Okey I solved it with pip install from the terminal with the env activated. From the nb it let me install but was not importing.
I have two questions. Do we ever normalise our embeddings, or is that useless since they anyway end up as input to the batch norm layer?
second is, how can we save these embeddings using Fastai?
1 Like
Hi Lawrie, are you installing v1 using a peterjc123 build? If so are you compiling yourself or is there a .whl available?
1 Like
I was building from source when the course first started however the pytorch team began running nightly builds for windows a few weeks later. Here is the current discussion. The peterjc123 builds seem to be failing at the moment but the most recent successful builds appear to be working fine. I am using the conda-3.6 cuda-90 build myself. You can download the tarball from his github page or follow the links to here to download directly.
Hope that helps.
2 Likes
I understand that in NLP and vision, we generally want to maximize the GPU memory and utilization during training. Is that idea generally the same for collaborative filtering and tabular data like the Rossmann competition, or do we often want to have small batch sizes even if they appear inefficient in terms of GPU utilization?
Yes, just pip install. No need for VS but you will need to pip install any dependencies first along with cuda if you are using a gpu.
Besides regularization affect, data augmentation also makes the CNN more robust by increasing its translation and scale invariance,
Yes, bias is also a “learned” parameter that is generally initialized as 0 or a small number in case of relu activations. Generally not randomly initialized as the random initialization of the weights suffices.
1 Like
I had a scenario for tabular data where all the features were continuous. So what feature engineering I can do to prepare such datasets.
@jeremy Could you turn Lesson 6 official resources and updates into wiki so others can contribute or edit? Thanks.
1 Like