I did a little bit of work on trying out SHARNN with fastai: Training SHA-RNN with fastai
Probably it might be useful 
2 Likes
Has anyone faced a dead kernel error while using the Tokenizer class?
Iâve a corpus of ~200,000 lines.
Steps to reproduce:
- Create
SentencePieceTokenizerclass - Pass in corpus into
SentencePieceTokenizer.setup(currently Iâm passing entire corpus because Iâm worried a subset might not capture all the possible words. The language Iâm using, has a very high morphology). - Pass SentencePieceTokenizer into Tokenizer in order to get FastAIâs additional tokenizer helper methods â
Here is my stack trace (Jupyter-Notebook log):
ntencepiece_trainer.cc(116) LOG(INFO) Running command: --input=tmp/texts.out --vocab_size=30000 --model_prefix=tmp/spm --character_coverage=0.99999 --model_type=unigram --unk_id=9 --pad_id=-1 --bos_id=-1 --eos_id=-1 --user_defined_symbols=âxxunk,âxxpad,âxxbos,âxxeos,âxxfld,âxxrep,âxxwrep,âxxup,âxxmaj
sentencepiece_trainer.cc(49) LOG(INFO) Starts training with :
TrainerSpec {
input: tmp/texts.out
input_format:
model_prefix: tmp/spm
model_type: UNIGRAM
vocab_size: 30000
self_test_sample_size: 0
character_coverage: 0.99999
input_sentence_size: 0
shuffle_input_sentence: 1
seed_sentencepiece_size: 1000000
shrinking_factor: 0.75
max_sentence_length: 4192
num_threads: 16
num_sub_iterations: 2
max_sentencepiece_length: 16
split_by_unicode_script: 1
split_by_number: 1
split_by_whitespace: 1
treat_whitespace_as_suffix: 0
user_defined_symbols: âxxunk
user_defined_symbols: âxxpad
user_defined_symbols: âxxbos
user_defined_symbols: âxxeos
user_defined_symbols: âxxfld
user_defined_symbols: âxxrep
user_defined_symbols: âxxwrep
user_defined_symbols: âxxup
user_defined_symbols: âxxmaj
hard_vocab_limit: 1
use_all_vocab: 0
unk_id: 9
bos_id: -1
eos_id: -1
pad_id: -1
unk_piece: <unk>
bos_piece: <s>
eos_piece: </s>
pad_piece: <pad>
unk_surface: â
}
NormalizerSpec {
name: nmt_nfkc
add_dummy_prefix: 1
remove_extra_whitespaces: 1
escape_whitespaces: 1
normalization_rule_tsv:
}
trainer_interface.cc(267) LOG(INFO) Loading corpus: tmp/texts.out
trainer_interface.cc(315) LOG(INFO) Loaded all 196848 sentences
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: âxxunk
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: âxxpad
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: âxxbos
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: âxxeos
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: âxxfld
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: âxxrep
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: âxxwrep
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: âxxup
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: âxxmaj
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: <unk>
trainer_interface.cc(335) LOG(INFO) Normalizing sentences...
trainer_interface.cc(384) LOG(INFO) all chars count=19916373
trainer_interface.cc(392) LOG(INFO) Done: 99.9991% characters are covered.
trainer_interface.cc(402) LOG(INFO) Alphabet size=94
trainer_interface.cc(403) LOG(INFO) Final character coverage=0.999991
trainer_interface.cc(435) LOG(INFO) Done! preprocessed 196846 sentences.
unigram_model_trainer.cc(129) LOG(INFO) Making suffix array...
unigram_model_trainer.cc(133) LOG(INFO) Extracting frequent sub strings...
unigram_model_trainer.cc(184) LOG(INFO) Initialized 793115 seed sentencepieces
trainer_interface.cc(441) LOG(INFO) Tokenizing input sentences with whitespace: 196846
trainer_interface.cc(451) LOG(INFO) Done! 577249
unigram_model_trainer.cc(470) LOG(INFO) Using 577249 sentences for EM training
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=0 size=280412 obj=16.2455 num_tokens=1181656 num_tokens/piece=4.214
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=1 size=236822 obj=13.6384 num_tokens=1185910 num_tokens/piece=5.0076
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=0 size=177327 obj=13.5918 num_tokens=1207058 num_tokens/piece=6.80696
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=1 size=175180 obj=13.5007 num_tokens=1208000 num_tokens/piece=6.89576
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=0 size=131345 obj=13.6065 num_tokens=1254268 num_tokens/piece=9.54942
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=1 size=131046 obj=13.5553 num_tokens=1254916 num_tokens/piece=9.57615
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=0 size=98280 obj=13.7175 num_tokens=1305316 num_tokens/piece=13.2816
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=1 size=98253 obj=13.6504 num_tokens=1305836 num_tokens/piece=13.2905
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=0 size=73688 obj=13.8502 num_tokens=1359889 num_tokens/piece=18.4547
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=1 size=73683 obj=13.7946 num_tokens=1360351 num_tokens/piece=18.4622
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=0 size=55262 obj=14.0404 num_tokens=1417469 num_tokens/piece=25.65
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=1 size=55260 obj=13.978 num_tokens=1418091 num_tokens/piece=25.6622
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=0 size=41445 obj=14.281 num_tokens=1481025 num_tokens/piece=35.7347
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=1 size=41445 obj=14.2049 num_tokens=1481276 num_tokens/piece=35.7408
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=0 size=33000 obj=14.4826 num_tokens=1543754 num_tokens/piece=46.7804
unigram_model_trainer.cc(486) LOG(INFO) EM sub_iter=1 size=33000 obj=14.401 num_tokens=1543804 num_tokens/piece=46.7819
trainer_interface.cc(507) LOG(INFO) Saving model: tmp/spm.model
trainer_interface.cc(531) LOG(INFO) Saving vocabs: tmp/spm.vocab
[I 19:20:37.324 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
WARNING:root:kernel f3fcecac-d891-4a93-bebb-368c3a1f809a restarted
Code
Tokenize using FastAI's SentencePiece
txts = L(o.open().read() for o in files[:1])
sp = SentencePieceTokenizer(lang='zul', vocab_sz=30000)
sp.setup(txts[:30000])
tkn = Tokenizer(sp)
type(tkn)
fastai2.text.core.Tokenizer
print(coll_repr(tkn(txt), 31))
Any and all help would be greatly appreciated.
cc @jeremy
FYI, fastai simply wraps SentencePiece. To debug this, first try doing it in regular SentencePiece to see if the issue is still there. This can be seen in the setup of SentencePieceProcessor:
def setup(self, items, rules=None):
from sentencepiece import SentencePieceProcessor
if rules is None: rules = []
if self.tok is not None: return {'sp_model': self.sp_model}
raw_text_path = self.cache_dir/'texts.out'
with open(raw_text_path, 'w') as f:
for t in progress_bar(maps(*rules, items), total=len(items), leave=False):
f.write(f'{t}\n')
sp_model = self.train(raw_text_path)
self.tok = SentencePieceProcessor() <-
self.tok.Load(str(sp_model))
def __call__(self, items):
for t in items: yield self.tok.EncodeAsPieces(t) <-
And all it does is use the SentencePiece library
Tokenizer itself is just a wrapper to get the tokenized words going
2 Likes
I used the SentencePiece library and no issues instead, it was the first thing I tried (Iâve spent 2.5 days stuck on just this first part). If possible I would like to skip the FastAI wrapper for SentencePiece and simply pass in, somehow, the output of sentencepiece to FastAI.
Can you suggest how I can go forward since Iâve already attempted the first solution you provided?
p.s. Iâve spent hours looking at the code â Iâve written python code for years and debugged many python apps as a seasoned engineer â thereâs something thatâs being implicit that should be explicit and thatâs what Iâm missing.
1 Like
Hi,
Iâm trying to understand the way the text dataloader work and how batches are divided as explained on chapter 12 of the amazing book.
I tried to create super simple dataset and inspect one batch from it:
test_L = L(((i,i+100) for i in range(100)))
bs = 5
test_chunks = group_chunks(test_L,bs)
print (coll_repr(test_L,10))
print (coll_repr(test_chunks,10))
cut = int(len(test_chunks) * 0.8)
dls_chunks = DataLoaders.from_dsets(
group_chunks(test_L[:cut], bs),
group_chunks(test_L[cut:], bs),
bs=bs, drop_last=True, shuffle=False)
x, y = dls_chunks.train.one_batch()
x[:10],y[:10],x.shape,y.shape,cut
and I donât understand the results :
(#100) [(0, 100),(1, 101),(2, 102),(3, 103),(4, 104),(5, 105),(6, 106),(7, 107),(8, 108),(9, 109)...]
(#100) [(0, 100),(20, 120),(40, 140),(60, 160),(80, 180),(1, 101),(21, 121),(41, 141),(61, 161),(81,
181)...]
(tensor([ 0, 16, 32, 48, 64, 1, 17, 33, 49, 65]),
tensor([100, 116, 132, 148, 164, 101, 117, 133, 149, 165]),
torch.Size([64]),
torch.Size([64]),
80)
- Why the length of the first batch I get from the one_batch is 64 (x and y) can I define the sequence length somehow?
- The order is not clear, wonât it be better for the training if each batch had continuous numbers?
1 Like
I can run the ULMFIT and IMDB notebook (fastai v0.0.11), but with my own data (Dutch Wikipedia with the same preparation as I use in fastai v1) I get the same error.
Do you have any thoughts on where I could start to debug?
cc @fmobrj75
@zerotosingularity what versions of PyTorch and torchvision? (Are they the most recent ones?)
1 Like
I tried 1.3.0 (with Torchvision 0.4) and 1.4.0 (with Torchvision 0.5)
Try to update both fastai2 and fastcore:
git clone https://github.com/fastai/fastai2
cd fastai2
conda env create -f environment.yml
source activate fastai2
python3 -m pip install packaging
git clone https://github.com/fastai/fastai2
cd fastai2
python3 -m pip install -e .[dev]
Then be sure fastcore is also up to date:
python3 -m pip uninstall fastcore
python3 -m pip install packaging
git clone https://github.com/fastai/fastcore
cd fastcore
python3 -m pip install -e .[dev]
Hope it also works for you.
1 Like
Thank you for the help!
I just tried these and I am now hitting an issue with Fastprogress (even trying to find the LR):
'NBMasterBar' object has no attribute 'out'
I have 0.2.3 installed and also tried the install from the repo.
Any chance you could list your environment so I can compare?
pip list
or
conda list -e
Hereâs mine:
This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
#_libgcc_mutex=0.1=main
absl-py=0.9.0=pypi_0
art=4.5=pypi_0
asn1crypto=1.3.0=py37_0
astor=0.8.1=pypi_0
attrs=19.3.0=py_0
backcall=0.1.0=py37_0
blas=1.0=mkl
bleach=3.1.0=py37_0
ca-certificates=2020.1.1=0
cachetools=4.0.0=pypi_0
certifi=2019.11.28=py37_0
cffi=1.14.0=py37h2e261b9_0
chardet=3.0.4=py37_1003
cmdstanpy=0.4.0=pypi_0
convertdate=2.2.0=pypi_0
coverage=5.0.3=pypi_0
cryptography=2.8=py37h1ba5d50_0
cudatoolkit=10.1.243=h6bb024c_0
cycler=0.10.0=py37_0
cymem=2.0.2=py37he1b5a44_0
cython=0.29.15=pypi_0
cython-blis=0.2.4=py37h516909a_1
dbus=1.13.12=h746ee38_0
decorator=4.4.2=py_0
defusedxml=0.6.0=py_0
entrypoints=0.3=py37_0
ephem=3.7.7.1=pypi_0
expat=2.2.6=he6710b0_0
fastai2=0.0.12=dev_0
fastcore=0.1.15=dev_0
fastprogress=0.2.3=dev_0
fastscript=0.1.4=pypi_0
fontconfig=2.13.0=h9420a91_0
freetype=2.9.1=h8a8886c_1
gast=0.2.2=pypi_0
glib=2.63.1=h5a9c865_0
gmp=6.1.2=h6c8ec71_1
google-auth=1.11.3=pypi_0
google-auth-oauthlib=0.4.1=pypi_0
google-pasta=0.2.0=pypi_0
grpcio=1.27.2=pypi_0
gst-plugins-base=1.14.0=hbbd80ab_1
gstreamer=1.14.0=hb453b48_1
h5py=2.10.0=pypi_0
holidays=0.10.1=pypi_0
icu=58.2=h9c2bf20_1
idna=2.8=py37_0
importlib_metadata=1.5.0=py37_0
intel-openmp=2020.0=166
ipykernel=5.1.4=py37h39e3cac_0
ipython=7.13.0=py37h5ca1d4c_0
ipython_genutils=0.2.0=py37_0
ipywidgets=7.5.1=py_0
jedi=0.16.0=py37_0
jinja2=2.11.1=py_0
joblib=0.14.1=py_0
jpeg=9b=h024ee3a_2
jsonschema=3.2.0=py37_0
jupyter=1.0.0=py37_7
jupyter_client=5.3.4=py37_0
jupyter_console=6.1.0=py_0
jupyter_core=4.6.1=py37_0
keras-applications=1.0.8=pypi_0
keras-preprocessing=1.1.0=pypi_0
kiwisolver=1.1.0=py37he6710b0_0
ld_impl_linux-64=2.33.1=h53a641e_7
libedit=3.1.20181209=hc058e9b_0
libffi=3.2.1=hd88cf55_4
libgcc-ng=9.1.0=hdf63c60_0
libgfortran-ng=7.3.0=hdf63c60_0
libpng=1.6.37=hbc83047_0
libsodium=1.0.16=h1bed415_0
libstdcxx-ng=9.1.0=hdf63c60_0
libtiff=4.1.0=h2733197_0
libuuid=1.0.3=h1bed415_2
libxcb=1.13=h1bed415_1
libxml2=2.9.9=hea5a465_1
lunarcalendar=0.0.9=pypi_0
markdown=3.2.1=pypi_0
markupsafe=1.1.1=py37h7b6447c_0
matplotlib=3.1.3=py37_0
matplotlib-base=3.1.3=py37hef1b27d_0
mistune=0.8.4=py37h7b6447c_0
mkl=2020.0=166
mkl-service=2.3.0=py37he904b0f_0
mkl_fft=1.0.15=py37ha843d7b_0
mkl_random=1.1.0=py37hd6b4f25_0
murmurhash=1.0.2=py37he6710b0_0
nbconvert=5.6.1=py37_0
nbdev=0.2.13=pypi_0
nbformat=5.0.4=py_0
ncurses=6.2=he6710b0_0
ninja=1.9.0=py37hfd86e86_0
notebook=6.0.3=py37_0
numpy=1.18.1=py37h4f9e942_0
numpy-base=1.18.1=py37hde5b4d6_1
oauthlib=3.1.0=pypi_0
olefile=0.46=py37_0
openssl=1.1.1d=h7b6447c_4
opt-einsum=3.2.0=pypi_0
packaging=20.3=pypi_0
pandas=0.25.3=pypi_0
pandoc=2.2.3.2=0
pandocfilters=1.4.2=py37_1
parso=0.6.1=py_0
pcre=8.43=he6710b0_0
pexpect=4.8.0=py37_0
pickleshare=0.7.5=py37_0
pillow=7.0.0=py37hb39fc2d_0
pip=20.0.2=py37_1
plac=0.9.6=py37_0
plotly=4.5.4=pypi_0
preshed=2.0.1=py37he6710b0_0
prometheus_client=0.7.1=py_0
prompt_toolkit=3.0.3=py_0
protobuf=3.11.3=pypi_0
ptyprocess=0.6.0=py37_0
pyasn1=0.4.8=pypi_0
pyasn1-modules=0.2.8=pypi_0
pycparser=2.19=py37_0
pygments=2.5.2=py_0
pymeeus=0.3.7=pypi_0
pyopenssl=19.1.0=py37_0
pyparsing=2.4.6=py_0
pyqt=5.9.2=py37h05f1152_2
pyrsistent=0.15.7=py37h7b6447c_0
pysocks=1.7.1=py37_0
pystan=2.19.1.1=pypi_0
python=3.7.6=h0371630_2
python-dateutil=2.8.1=py_0
pytorch=1.4.0=py3.7_cuda10.1.243_cudnn7.6.3_0
pytz=2019.3=py_0
pyyaml=5.3=py37h7b6447c_0
pyzmq=18.1.1=py37he6710b0_0
qt=5.9.7=h5867ecd_1
qtconsole=4.7.1=py_0
qtpy=1.9.0=py_0
readline=7.0=h7b6447c_5
requests=2.23.0=py37_0
requests-oauthlib=1.3.0=pypi_0
retrying=1.3.3=pypi_0
rsa=4.0=pypi_0
scikit-learn=0.22.1=py37hd81dba3_0
scipy=1.4.1=py37h0b6359f_0
seaborn=0.10.0=pypi_0
send2trash=1.5.0=py37_0
setuptools=46.0.0=py37_0
setuptools-git=1.2=pypi_0
sip=4.19.8=py37hf484d3e_0
six=1.14.0=py37_0
spacy=2.1.8=py37hc9558a2_0
sqlite=3.31.1=h7b6447c_0
srsly=0.1.0=py37he1b5a44_0
tensorboard=2.1.1=pypi_0
tensorflow=2.1.0=pypi_0
tensorflow-estimator=2.1.0=pypi_0
termcolor=1.1.0=pypi_0
terminado=0.8.3=py37_0
testpath=0.4.4=py_0
thinc=7.0.8=py37hc9558a2_0
tk=8.6.8=hbc83047_0
torchvision=0.5.0=py37_cu101
tornado=6.0.4=py37h7b6447c_1
tqdm=4.42.1=py_0
traitlets=4.3.3=py37_0
tscv=0.0.4=pypi_0
urllib3=1.25.8=py37_0
wasabi=0.2.2=py_0
wcwidth=0.1.8=py_0
webencodings=0.5.1=py37_1
werkzeug=1.0.0=pypi_0
wheel=0.34.2=py37_0
widgetsnbextension=3.5.1=py37_0
wrapt=1.12.1=pypi_0
xz=5.2.4=h14c3975_4
yaml=0.1.7=had09818_2
zeromq=4.3.1=he6710b0_3
zipp=2.2.0=py_0
zlib=1.2.11=h7b6447c_3
zstd=1.3.7=h0b5b093_0
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
_libgcc_mutex=0.1=main
alpha-vantage=2.1.3=pypi_0
asn1crypto=1.3.0=py37_0
attrs=19.3.0=py_0
backcall=0.1.0=py37_0
beautifulsoup4=4.8.2=pypi_0
blas=1.0=mkl
bleach=3.1.0=py_0
bs4=0.0.1=pypi_0
ca-certificates=2020.1.1=0
certifi=2019.11.28=py37_0
cffi=1.14.0=py37h2e261b9_0
chardet=3.0.4=py37_1003
cryptography=2.8=py37h1ba5d50_0
cudatoolkit=10.1.243=h6bb024c_0
cycler=0.10.0=py37_0
cymem=2.0.2=py37he1b5a44_0
cython-blis=0.2.4=py37h516909a_1
dbus=1.13.12=h746ee38_0
decorator=4.4.1=py_0
defusedxml=0.6.0=py_0
entrypoints=0.3=py37_0
expat=2.2.6=he6710b0_0
fast-tabnet=0.0.5=pypi_0
fastai2=0.0.8=dev_0
fastcore=0.1.11=dev_0
fastprogress=0.2.2=py_0
fastscript=0.1.4=pypi_0
fontconfig=2.13.0=h9420a91_0
freetype=2.9.1=h8a8886c_1
glib=2.63.1=h5a9c865_0
gmp=6.1.2=h6c8ec71_1
gst-plugins-base=1.14.0=hbbd80ab_1
gstreamer=1.14.0=hb453b48_1
icu=58.2=h9c2bf20_1
idna=2.8=py37_0
importlib_metadata=1.5.0=py37_0
intel-openmp=2020.0=166
ipykernel=5.1.4=py37h39e3cac_0
ipython=7.12.0=py37h5ca1d4c_0
ipython_genutils=0.2.0=py37_0
ipywidgets=7.5.1=py_0
jedi=0.16.0=py37_0
jinja2=2.11.1=py_0
joblib=0.14.1=py_0
jpeg=9b=h024ee3a_2
jsonschema=3.2.0=py37_0
jupyter=1.0.0=py37_7
jupyter_client=5.3.4=py37_0
jupyter_console=6.1.0=py_0
jupyter_core=4.6.1=py37_0
kiwisolver=1.1.0=py37he6710b0_0
ld_impl_linux-64=2.33.1=h53a641e_7
libedit=3.1.20181209=hc058e9b_0
libffi=3.2.1=hd88cf55_4
libgcc-ng=9.1.0=hdf63c60_0
libgfortran-ng=7.3.0=hdf63c60_0
libpng=1.6.37=hbc83047_0
libsodium=1.0.16=h1bed415_0
libstdcxx-ng=9.1.0=hdf63c60_0
libtiff=4.1.0=h2733197_0
libuuid=1.0.3=h1bed415_2
libxcb=1.13=h1bed415_1
libxml2=2.9.9=hea5a465_1
lxml=4.5.0=pypi_0
markupsafe=1.1.1=py37h7b6447c_0
matplotlib=3.1.3=py37_0
matplotlib-base=3.1.3=py37hef1b27d_0
mdlp-discretization=0.3.3=pypi_0
mistune=0.8.4=py37h7b6447c_0
mkl=2020.0=166
mkl-service=2.3.0=py37he904b0f_0
mkl_fft=1.0.15=py37ha843d7b_0
mkl_random=1.1.0=py37hd6b4f25_0
more-itertools=8.2.0=pypi_0
murmurhash=1.0.2=py37he6710b0_0
nbconvert=5.6.1=py37_0
nbdev=0.2.12=pypi_0
nbformat=5.0.4=py_0
ncurses=6.1=he6710b0_1
ninja=1.9.0=py37hfd86e86_0
notebook=6.0.3=py37_0
numpy=1.18.1=py37h4f9e942_0
numpy-base=1.18.1=py37hde5b4d6_1
olefile=0.46=py_0
openssl=1.1.1d=h7b6447c_4
packaging=20.1=pypi_0
pandas=1.0.1=py37h0573a6f_0
pandoc=2.2.3.2=0
pandocfilters=1.4.2=py37_1
parso=0.6.1=py_0
pcre=8.43=he6710b0_0
pexpect=4.8.0=py37_0
pickleshare=0.7.5=py37_0
pillow=7.0.0=py37hb39fc2d_0
pip=20.0.2=py37_1
plac=0.9.6=py37_0
pluggy=0.13.1=pypi_0
preshed=2.0.1=py37he6710b0_0
prometheus_client=0.7.1=py_0
prompt_toolkit=3.0.3=py_0
psycopg2=2.8.4=pypi_0
ptyprocess=0.6.0=py37_0
py=1.8.1=pypi_0
pycparser=2.19=py_0
pygments=2.5.2=py_0
pyopenssl=19.1.0=py37_0
pyparsing=2.4.6=py_0
pyqt=5.9.2=py37h05f1152_2
pyrsistent=0.15.7=py37h7b6447c_0
pysocks=1.7.1=py37_0
pytest=5.3.5=pypi_0
python=3.7.6=h0371630_2
python-dateutil=2.8.1=py_0
pytorch=1.3.1=py3.7_cuda10.1.243_cudnn7.6.3_0
pytz=2019.3=py_0
pyyaml=5.3=py37h7b6447c_0
pyzmq=18.1.1=py37he6710b0_0
qt=5.9.7=h5867ecd_1
qtconsole=4.6.0=py_1
readline=7.0=h7b6447c_5
requests=2.22.0=py37_1
scikit-learn=0.22.1=py37hd81dba3_0
scipy=1.4.1=py37h0b6359f_0
send2trash=1.5.0=py37_0
sentencepiece=0.1.85=pypi_0
setuptools=45.2.0=py37_0
sip=4.19.8=py37hf484d3e_0
six=1.14.0=py37_0
soupsieve=1.9.5=pypi_0
spacy=2.1.8=py37hc9558a2_0
sqlalchemy=1.3.14=pypi_0
sqlite=3.31.1=h7b6447c_0
srsly=0.1.0=py37he1b5a44_0
terminado=0.8.3=py37_0
testpath=0.4.4=py_0
thinc=7.0.8=py37hc9558a2_0
tk=8.6.8=hbc83047_0
torchvision=0.4.2=py37_cu101
tornado=6.0.3=py37h7b6447c_3
tqdm=4.42.1=py_0
traitlets=4.3.3=py37_0
urllib3=1.25.8=py37_0
wasabi=0.2.2=py_0
wcwidth=0.1.8=py_0
webencodings=0.5.1=py37_1
wheel=0.34.2=py37_0
widgetsnbextension=3.5.1=py37_0
xlrd=1.2.0=pypi_0
xz=5.2.4=h14c3975_4
yaml=0.1.7=had09818_2
zeromq=4.3.1=he6710b0_3
zipp=2.2.0=py_0
zlib=1.2.11=h7b6447c_3
zstd=1.3.7=h0b5b093_0Another issue I ran into some point in time (I dont remember when) that maybe you will face. Some point in time, I had CUDA 9 installed, so I had to migrate to Cuda 10 to make new versions of pytorch work.
Yours:
Might need to stick with Fastai2 v 0.0.11 for now. That one runs the IMBD and ULMFIT notebook, but seems to not work with my data (which used to work in v1).
I already cut of texts that were shorter than 5/10 words, but to no availâŚ
Thank you for the help! Iâll continue my searchâŚ
Have you checked your CUDA version? If suspect pytorch 1.4 requires CUDA 10.
Iâm running Cuda 10.2 and the latest Nvidia drivers.
From the current course info, I see we will likely start with 0.0.11 instead of the development version. I will start with that one, and try to find a solution. On 0.0.11 the IMDB and ULMFIT notebooks seem to workâŚ
Iâm having some errors that might be related.
Iâm following the following notebooks, not sure which one I should focus on, but they are all very similar:
https://github.com/fastai/fastbook/blob/master/10_nlp.ipynb
https://github.com/fastai/fastai2/blob/master/nbs/course/lesson3-imdb.ipynb
https://github.com/fastai/course-v4/blob/master/nbs/10_nlp.ipynb
After successfully training a language model, Iâm unable to use learn.predict to predict the next word:
TEXT = "I liked this movie because"
N_WORDS = 40
N_SENTENCES = 2
print("\n".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))
Error:
AttributeError Traceback (most recent call last)
<ipython-input-21-135dd315cd3b> in <module>
----> 1 print("\n".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))
<ipython-input-21-135dd315cd3b> in <genexpr>(.0)
----> 1 print("\n".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))
~/git_packages/fastai2/fastai2/text/learner.py in predict(self, text, n_words, no_unk, temperature, min_p, rm_type_tfms, no_bar, decoder, **kwargs)
127 tokens = [tfm.vocab[i] for i in idxs if tfm.vocab[i] not in [BOS, PAD]]
128 sep = self.dls.train_ds.tokenizer
--> 129 return sep.join(decoder(tokens))
130
131 @delegates(Learner.get_preds)
AttributeError: 'L' object has no attribute 'join'
Iâm able to continue on, but when I get to the step to train the classifier, I get another AssertionError:
imdb_clas = DataBlock(blocks=(TextBlock.from_df('letters', vocab=dbunch.vocab), MultiCategoryBlock),
get_x=attrgetter('text'),
splitter=RandomSplitter(),
get_y=ColReader(13, label_delim=','))
dbunch_class = imdb_clas.dataloaders(df_all2, bs=8, seq_len=80)
learn_class = text_classifier_learner(dbunch_class, AWD_LSTM, drop_mult=0.5, metrics=accuracy).to_fp16()
learn_class = learn_class.load_encoder('finetuned')
learn_class.fit_one_cycle(1, 1e-2, moms=(0.8,0.7, 0.8))
Hereâs the error:
AssertionError Traceback (most recent call last)
<ipython-input-93-2885a81a7141> in <module>
----> 1 learn_class.fit_one_cycle(1, 1e-2, moms=(0.8,0.7, 0.8))
~/git_packages/fastai2/fastai2/callback/schedule.py in fit_one_cycle(self, n_epoch, lr_max, div, div_final, pct_start, wd, moms, cbs, reset_opt)
110 scheds = {'lr': combined_cos(pct_start, lr_max/div, lr_max, lr_max/div_final),
111 'mom': combined_cos(pct_start, *(self.moms if moms is None else moms))}
--> 112 self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd)
113
114 # Cell
~/git_packages/fastai2/fastai2/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt)
189 self.epoch=epoch; self('begin_epoch')
190 self._do_epoch_train()
--> 191 self._do_epoch_validate()
192 except CancelEpochException: self('after_cancel_epoch')
193 finally: self('after_epoch')
~/git_packages/fastai2/fastai2/learner.py in _do_epoch_validate(self, ds_idx, dl)
171 dl,old,has = change_attrs(dl, names, [False,False])
172 self.dl = dl; self('begin_validate')
--> 173 with torch.no_grad(): self.all_batches()
174 except CancelValidException: self('after_cancel_validate')
175 finally:
~/git_packages/fastai2/fastai2/learner.py in all_batches(self)
139 def all_batches(self):
140 self.n_iter = len(self.dl)
--> 141 for o in enumerate(self.dl): self.one_batch(*o)
142
143 def one_batch(self, i, b):
~/git_packages/fastai2/fastai2/learner.py in one_batch(self, i, b)
153 self.opt.zero_grad()
154 except CancelBatchException: self('after_cancel_batch')
--> 155 finally: self('after_batch')
156
157 def _do_begin_fit(self, n_epoch):
~/git_packages/fastai2/fastai2/learner.py in __call__(self, event_name)
120 def ordered_cbs(self, cb_func): return [cb for cb in sort_by_run(self.cbs) if hasattr(cb, cb_func)]
121
--> 122 def __call__(self, event_name): L(event_name).map(self._call_one)
123 def _call_one(self, event_name):
124 assert hasattr(event, event_name)
~/git_packages/fastcore/fastcore/foundation.py in map(self, f, *args, **kwargs)
360 else f.format if isinstance(f,str)
361 else f.__getitem__)
--> 362 return self._new(map(g, self))
363
364 def filter(self, f, negate=False, **kwargs):
~/git_packages/fastcore/fastcore/foundation.py in _new(self, items, *args, **kwargs)
313 @property
314 def _xtra(self): return None
--> 315 def _new(self, items, *args, **kwargs): return type(self)(items, *args, use_list=None, **kwargs)
316 def __getitem__(self, idx): return self._get(idx) if is_indexer(idx) else L(self._get(idx), use_list=None)
317 def copy(self): return self._new(self.items.copy())
~/git_packages/fastcore/fastcore/foundation.py in __call__(cls, x, *args, **kwargs)
39 return x
40
---> 41 res = super().__call__(*((x,) + args), **kwargs)
42 res._newchk = 0
43 return res
~/git_packages/fastcore/fastcore/foundation.py in __init__(self, items, use_list, match, *rest)
304 if items is None: items = []
305 if (use_list is not None) or not _is_array(items):
--> 306 items = list(items) if use_list else _listify(items)
307 if match is not None:
308 if is_coll(match): match = len(match)
~/git_packages/fastcore/fastcore/foundation.py in _listify(o)
240 if isinstance(o, list): return o
241 if isinstance(o, str) or _is_array(o): return [o]
--> 242 if is_iter(o): return list(o)
243 return [o]
244
~/git_packages/fastcore/fastcore/foundation.py in __call__(self, *args, **kwargs)
206 if isinstance(v,_Arg): kwargs[k] = args.pop(v.i)
207 fargs = [args[x.i] if isinstance(x, _Arg) else x for x in self.pargs] + args[self.maxi+1:]
--> 208 return self.fn(*fargs, **kwargs)
209
210 # Cell
~/git_packages/fastai2/fastai2/learner.py in _call_one(self, event_name)
123 def _call_one(self, event_name):
124 assert hasattr(event, event_name)
--> 125 [cb(event_name) for cb in sort_by_run(self.cbs)]
126
127 def _bn_bias_state(self, with_bias): return bn_bias_params(self.model, with_bias).map(self.opt.state)
~/git_packages/fastai2/fastai2/learner.py in <listcomp>(.0)
123 def _call_one(self, event_name):
124 assert hasattr(event, event_name)
--> 125 [cb(event_name) for cb in sort_by_run(self.cbs)]
126
127 def _bn_bias_state(self, with_bias): return bn_bias_params(self.model, with_bias).map(self.opt.state)
~/git_packages/fastai2/fastai2/callback/core.py in __call__(self, event_name)
21 _run = (event_name not in _inner_loop or (self.run_train and getattr(self, 'training', True)) or
22 (self.run_valid and not getattr(self, 'training', False)))
---> 23 if self.run and _run: getattr(self, event_name, noop)()
24 if event_name=='after_fit': self.run=True #Reset self.run to True at each end of fit
25
~/git_packages/fastai2/fastai2/learner.py in after_batch(self)
406 if len(self.yb) == 0: return
407 mets = self._train_mets if self.training else self._valid_mets
--> 408 for met in mets: met.accumulate(self.learn)
409 if not self.training: return
410 self.lrs.append(self.opt.hypers[-1]['lr'])
~/git_packages/fastai2/fastai2/learner.py in accumulate(self, learn)
331 def accumulate(self, learn):
332 bs = find_bs(learn.yb)
--> 333 self.total += to_detach(self.func(learn.pred, *learn.yb))*bs
334 self.count += bs
335 @property
~/git_packages/fastai2/fastai2/metrics.py in accuracy(inp, targ, axis)
72 def accuracy(inp, targ, axis=-1):
73 "Compute accuracy with `targ` when `pred` is bs * n_classes"
---> 74 pred,targ = flatten_check(inp.argmax(dim=axis), targ)
75 return (pred == targ).float().mean()
76
~/git_packages/fastai2/fastai2/torch_core.py in flatten_check(inp, targ)
763 "Check that `out` and `targ` have the same number of elements and flatten them."
764 inp,targ = inp.contiguous().view(-1),targ.contiguous().view(-1)
--> 765 test_eq(len(inp), len(targ))
766 return inp,targ
~/git_packages/fastcore/fastcore/test.py in test_eq(a, b)
30 def test_eq(a,b):
31 "`test` that `a==b`"
---> 32 test(a,b,equals, '==')
33
34 # Cell
~/git_packages/fastcore/fastcore/test.py in test(a, b, cmp, cname)
20 "`assert` that `cmp(a,b)`; display inputs and `cname or cmp.__name__` if it fails"
21 if cname is None: cname=cmp.__name__
---> 22 assert cmp(a,b),f"{cname}:\n{a}\n{b}"
23
24 # Cell
AssertionError: ==:
8
24
Note, this trains fine when I donât use a MultiCategoryBlock:
imdb_clas = DataBlock(blocks=(TextBlock.from_df('letters', vocab=dbunch.vocab), CategoryBlock),
get_x=attrgetter('text'),
get_y=attrgetter('type'),
splitter=RandomSplitter())
Iâm on fastcore version: 0.1.16, and fastai2 version 0.0.13
Thanks!
You can[t use accuracy for a multi-label problem (thatâs why you get the second error). Use accuracy_thresh.
1 Like
Ahh, thatâs right!
I couldnât get it to work with accuracy_thresh, but it works and Iâm able to train with Perplexity() or simply dropping the metrics part completely:
learn_class = text_classifier_learner(dbunch_class, AWD_LSTM, drop_mult=0.5).to_fp16()
learn_class = learn_class.load_encoder('finetuned')
Thanks!
Oh sorry, accuracy_multi is the name in v2 (accuracy_thresh was in v1).
3 Likes
That does indeed work! Thanks!!