SentencePiece

Daniel.R.Armstrong · September 18, 2019, 6:56pm

I am glad you found it, if you want to save the training time feel free to use the one I trained.

rahuluppari · July 22, 2020, 10:37pm

Hello Daniel,

Could you please help me to solve an error which im facing during the downloading the get_wiki

1: I’m not able to use nlputils it is triggering an error.
from nlputils import get_wiki,split_wiki
no module named ‘nlputils’

While using get_wiki(lang,path) it is giving the below error:
[Errno 2] No such file or directory: ‘C:\Users\NANI.HOMIES\.fastai\data\mrwiki\text\AA\wiki_00’

Could you please help me with solving this error.

thanking you in advance

rahuluppari · July 22, 2020, 10:39pm

Hello Jeremy,

Could you please help me to solve an error which im facing during the downloading the get_wiki

1: I’m not able to use nlputils it is triggering an error.
from nlputils import get_wiki,split_wiki
no module named ‘nlputils’

While using get_wiki(lang,path) it is giving the below error:
[Errno 2] No such file or directory: ‘C:\Users\NANI.HOMIES.fastai\data\mrwiki\text\AA\wiki_00’

Could you please help me with solving this error.

thanking you in advance

mrfabulous1 · July 23, 2020, 10:30am

Hi ahuluppari hope you are having a wonderful day!

from nlputils import get_wiki,split_wiki
no module named ‘nlputils’

Normally the error above means that you haven’t installed the library.

run the following pip list to check if the library is installed.

Use pip install to install the library.

[quote=“rahuluppari, post:35, topic:53010”]
While using get_wiki(lang,path) it is giving the below error:
[Errno 2] No such file or directory: ‘C:\Users\NANI.HOMIES.fastai\data\mrwiki\text\AA\wiki_00’
[/quote

I don’t think its possible for this to work as it looks like it is being called by the import statement before it.

Hope this helps.

Cheers mrfabulous1 :smiley

ps. we don’t normally ping Jeremy for such an issue

rahuluppari · July 23, 2020, 10:55am

Hello mrfabulous1,

Many Thanks for your response.
Apologies for the same!!

I have the list of libraries installed and nlputils was installed, so i have uninstalled it and reinstalled the library “pip install nlputils” but it is still giving the same error.
no module named “nlputils”.

I don’t know the reason why

I have uploaded the snapshot for the same for your reference!!

It would be a great help if you can help me to resolve this issue.

Daniel.R.Armstrong · July 23, 2020, 5:22pm

@rahuluppari Is the nlputis.py file in the same directory as the notebook you are running? If you haven’t done so already I would watch the Turkish lesson. Then watch it again. Then maybe one more time.

rahuluppari · July 24, 2020, 12:02am

Many thanks @Daniel.R.Armstrong,

It worked, I didn’t saved the nlputils.py file in my working directory.

@Daniel.R.Armstrong, I do still have problem with get_wiki(path,lang) it is giving an error
below are the snapshot for the same

It would a great help if you could help me to solve this error as well.

Thanking you in advance!!!

Daniel.R.Armstrong · July 24, 2020, 12:39am

@rahuluppari, I think that you need to make sure that the wiki downloads are in that location. I am sure that there are lots of ways to do it, but I use a config file to make sure everything is where I expected it to be.

rahuluppari · July 24, 2020, 9:10am

@Daniel.R.Armstrong, thanks for your reply.

Even I’m using config file for wiki downloads, still it is giving me the error.
below is the snapshot of the downloads.

Anyother way to solve this error.

rahuluppari · July 24, 2020, 9:11am

@Daniel.R.Armstrong

Even i tried doing the same thing in Google Colab.
It is giving me the same error.

Daniel.R.Armstrong · July 24, 2020, 2:04pm

@rahuluppari, If i was you, I would clone the fastai nlp repo and run the vietnamese notebook while you are following along with lesson 10 , to make sure it works. After that do the same with the Turkish notebook. After that you can make changes to fit your needs.

rahuluppari · July 24, 2020, 2:57pm

Thanks @Daniel.R.Armstrong,

I have one more small, it might sound silly.

I have no idea from where I can get the fastai nlp repo.

Daniel.R.Armstrong · July 24, 2020, 4:30pm

rahuluppari · July 25, 2020, 5:21pm

Many thanks @Daniel.R.Armstrong

Could you help me understand one more thing. where should i clone the repository should i clone it through command or should i download zip folder from git hub and store it.

but to which location?

Daniel.R.Armstrong · July 25, 2020, 8:56pm

I didnt follow these instructions but they look pretty good.

Daniel.R.Armstrong · July 25, 2020, 9:05pm

@rahuluppari if you want a different perspective on the fastai nlp class, this was a study group. https://www.youtube.com/playlist?list=PLILZm3MRkvH_Yf4Ah9pkxgyzo9oi3mEly

rahuluppari · July 27, 2020, 8:03pm

Many thanks @Daniel.R.Armstrong.

Even after trying these steps it is still giving the path error.

If i use the same in linux will it make any difference and able to help me execute my code.

Regards,
Rahul Ramchandra Uppari.

Daniel.R.Armstrong · July 28, 2020, 3:07am

@rahuluppari , I had issues with cofig files and path defaults in fast.ai, when I started using it. You need to inspect the path variables, and make sure you look at the each step. Sometimes I break apart a function and look what is happening step by step. I would also delete the wiki folder then try again.

It is complete junk, I didn’t know what I was doing, when I did it, but this is the notebook that I used to do it for french on GCP. Keep in mind that I didn’t run the code top to bottom.

github.com

Daniel-R-Armstrong/French-Wiki-2500-Pretrained-SentencePiece-LM/blob/master/fr_spm.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "from fastai import *\n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},

This file has been truncated. show original

wjs20 · November 7, 2020, 10:07am

Hi Daniel

I have trained a sentencepiece tokenizer and have the .vocab and .model files in a tmp folder. If I wanted to use them again How would I pass these into fastai2’s datablock?

Thanks!

Daniel.R.Armstrong · November 7, 2020, 3:16pm

I haven’t had the time use SentencePiece in v2, sorry!