dl = LMDataLoader(nums200)
from notebook lesson 10
after this our dataloder is prepared
x,y = first(dl)
x.shape,y.shape
but calling learner(dl)
‘list’ object has no attribute ‘train’
so how to use this dl with learner
Hi @anishjain,
If you look a few cells lower, the processing is made for the full dataset here :
get_imdb = partial(get_text_files, folders=[‘train’, ‘test’, ‘unsup’])
dls_lm = DataBlock(
blocks=TextBlock.from_folder(path, is_lm=True),
get_items=get_imdb, splitter=RandomSplitter(0.1)
).dataloaders(path, path=path, bs=128, seq_len=80)
You start by creating a data block that gets converted into dataloaders. In your case you only have a data loader and it does not know what is used to train and what is used to validate.
Hope it helps !
Charles
yes thanks but i forget to mention this point i already see what u suggest it works. So we create the previous one only for learning how tokenization and numecalization works . so lets say data is not arranged in this IMDB format and arrange like some text in CSV file some in .txt files so how to this approach “DataBlock” in this case previous one works well unless we make learner quite confussed