we must be sampling without replacement
Well, doesn’t it get the whole batch into memory? What if more than one row of a batch can’t fit in memory?
A link has been posted twice to answer that question. Please click on it before asking it again 
I have put it up in the wiki.
1 Like
Now, you only collate the samples you want to yield right now. If your batch size is 64, just those 64 samples.
1 Like
Note that num_workers > 1 could bring you some problems with memory leakage. Not sure if that was somehow fixed in the most recent PyTorch or if it’s an essential Python’s issue.
7 Likes
A lot times in kaggle and colab kernels. (a very big headache)
1 Like
How do we incrementally train the model? Do we use the parameters of the original training data i.e. mean and std dev and keep training it?
thanks…
What does
With torch.no_Grad ensures ?
@champs.jaideep It ensures that you do not perform backprop on the validation set in this case…
1 Like
If I can ask a bit of an off topic question. Do we anticipate more integration of generative models in fast.ai soon? Or is it that the main components are given and it’s up to us to construct the more complex models?
When you do the forward pass, you need to store intermediary results for the backward pass (as we have shown in notebook 02). Using with torch.no_grad removes that default behavior to let you save GPU memory.
3 Likes
I would also love to see some of this stuff done on sequence models…
If i understand the initialization schemes correctly, it is all so that we don’t need regularization when training the network. Does this mean we don’t need dropout if we do proper initalization either?
1 Like
The goal of part 2 is to teach you how to build your own thing that fits the task you are tackling. We have the integrated tools for mainstream tasks and part one was about those. We’ll never get all the tools for specific tasks so we have chosen to focus on making the library highly flexible, then explain to all of you how you can customize it to your needs.
3 Likes
I am not sure if you don’t need it due to initialization alone but I do know that dropout has fallen out of favor and BN is used instead.
Similar question. This looks like general approach for most models. Is there any reason why this is only applied on ConvNets?
Got it thanks, makes sense!
Not an expert but have been working with generative models a lot lately, will give it a shot to do fast.ai versions.
1 Like
Initialization isn’t intended to remove the need for regularization. There are papers that show you can train effectively without regularization using good initialization, but removing regularization isn’t the point.
No it’s very different. if you don’t initialize properly, you have a chance that your model just won’t train. No matter what amount of regularization you have.
1 Like