Thanks! I’m confused, though, because there is not a one-to-one correlation between the contents of !pip list from Gradient/Jupyter and what’s in the Github repo now. In fact, there are certain elements in the Github repo requirements file that are nowhere to be found in the pip list.
Here’s the requirements.txt:
aiofiles==0.4.0
aiohttp==3.5.4
asyncio==3.4.3
fastai==1.0.60
https://download.pytorch.org/whl/cpu/torch-1.1.0-cp37-cp37m-linux_x86_64.whl 4
https://download.pytorch.org/whl/cpu/torchvision-0.3.0-cp37-cp37m-linux_x86_64.whl
numpy==1.16.3
pillow~=6.0
python-multipart==0.0.5
starlette==0.12.0
uvicorn==0.7.1
And here’s the !pip list:
asn1crypto 0.24.0
attrs 18.2.0
backcall 0.1.0
beautifulsoup4 4.7.1
bleach 3.1.0
Bottleneck 1.2.1
certifi 2018.11.29
cffi 1.11.5
chardet 3.0.4
conda 4.8.1
conda-package-handling 1.6.0
cryptography 2.3.1
cycler 0.10.0
cymem 2.0.2
cytoolz 0.9.0.1
dataclasses 0.6
decorator 4.3.0
dill 0.2.8.2
entrypoints 0.3
fastai 1.0.60
fastprogress 0.2.2
idna 2.8
ipykernel 5.1.0
ipython 7.2.0
ipython-genutils 0.2.0
ipywidgets 7.4.2
jedi 0.13.2
Jinja2 2.10
jsonschema 3.0.0a3
jupyter 1.0.0
jupyter-client 5.2.4
jupyter-console 6.0.0
jupyter-core 4.4.0
kiwisolver 1.0.1
MarkupSafe 1.1.0
matplotlib 3.0.2
mistune 0.8.4
mkl-fft 1.0.10
mkl-random 1.0.2
msgpack 0.6.0
msgpack-numpy 0.4.3.2
murmurhash 1.0.0
nb-conda 2.2.1
nb-conda-kernels 2.2.0
nbconvert 5.3.1
nbformat 4.4.0
notebook 5.7.4
numexpr 2.6.9
numpy 1.15.4
nvidia-ml-py3 7.352.0
olefile 0.46
packaging 19.0
pandas 0.23.4
pandocfilters 1.4.2
parso 0.3.1
pexpect 4.6.0
pickleshare 0.7.5
Pillow 5.4.1
pip 18.1
plac 0.9.6
preshed 2.0.1
prometheus-client 0.5.0
prompt-toolkit 2.0.7
ptyprocess 0.6.0
pycosat 0.6.3
pycparser 2.19
Pygments 2.3.1
pyOpenSSL 18.0.0
pyparsing 2.3.1
pyrsistent 0.14.9
PySocks 1.6.8
python-dateutil 2.7.5
pytz 2018.9
PyYAML 3.13
pyzmq 17.1.2
qtconsole 4.4.3
regex 2018.1.10
requests 2.21.0
ruamel-yaml 0.15.87
scipy 1.2.0
Send2Trash 1.5.0
setuptools 40.6.3
six 1.12.0
soupsieve 1.7.1
spacy 2.0.18
terminado 0.8.1
testpath 0.4.2
thinc 6.12.1
toolz 0.9.0
torch 1.4.0
torchvision 0.5.0
tornado 5.1.1
tqdm 4.29.1
traitlets 4.3.2
typing 3.6.4
ujson 1.35
urllib3 1.24.1
wcwidth 0.1.7
webencodings 0.5.1
wheel 0.32.3
widgetsnbextension 3.4.2
wrapt 1.10.11
By the way, I did try to set up a virtual env on my own machine, but I hit a problem with that as well. Life is rough for us newbies!
Thanks!
HI cberkner Hope you are having a wonderul day!
No need to be confused Python on Gradient has many libraries so when ever a newbie or experienced person logs in, they do not have to install every library themselves.
You are only concerned about the libraries in the repo, or any additional libraries your app uses. For example your repo has incorrect versions for the two libraries below.
As mentioned in other posts you need to change all the repo library versions to match the gradient versions you are using. Replace the two pytorch libraries as they are not the versions you trained your model on using Gradient (see your pip list). Also change any other libraries that do not match.
If your model is old I would train it again as I consider pip list and the model (.pkl file) a pair. You must do the two together just in case library changes are made by the owners.
What problem did you have when you tried a standalone virtual env?
Problems are good, which is why Jeremy wants us to go from model to application as it makes you learn loads of stuff. I had about forty problems on my first classifier from docker.com to github.com and it took about 2-4 weeks from watching the videos to deployment. Lol! 
Now I can deploy a simple classifier in about 15 - 30 mins.
Cheers mrfabulous1
Thanks, but unfortunately it’s still not working after updating all of the library versions. My model is just a week old.
Here is the error output from Render:
Jun 19 03:47:15 PM #10 2.951 Traceback (most recent call last):
Jun 19 03:47:15 PM #10 2.951 File “app/server.py”, line 48, in
Jun 19 03:47:15 PM #10 2.951 learn = loop.run_until_complete(asyncio.gather(*tasks))[0]
Jun 19 03:47:15 PM #10 2.951 File “/usr/local/lib/python3.7/asyncio/base_events.py”, line 587, in run_until_complete
Jun 19 03:47:15 PM #10 2.951 return future.result()
Jun 19 03:47:15 PM #10 2.951 File “app/server.py”, line 35, in setup_learner
Jun 19 03:47:15 PM #10 2.951 learn = load_learner(path, export_file_name)
Jun 19 03:47:15 PM #10 2.951 File “/usr/local/lib/python3.7/site-packages/torch/serialization.py”, line 529, in load
Jun 19 03:47:15 PM #10 2.951 File “/usr/local/lib/python3.7/site-packages/fastai/basic_train.py”, line 618, in load_learner
Jun 19 03:47:15 PM #10 2.951 state = torch.load(source, map_location=‘cpu’) if defaults.device == torch.device(‘cpu’) else torch.load(source)
Jun 19 03:47:15 PM #10 2.951 magic_number = pickle_module.load(f, **pickle_load_args)
Jun 19 03:47:15 PM #10 2.951 return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
Jun 19 03:47:15 PM #10 2.951 File “/usr/local/lib/python3.7/site-packages/torch/serialization.py”, line 692, in _legacy_load
Jun 19 03:47:16 PM #10 ERROR: executor failed running [/bin/sh -c python app/server.py]: buildkit-runc did not terminate successfully
Jun 19 03:47:15 PM #10 2.951 _pickle.UnpicklingError: invalid load key, ‘<’.
Jun 19 03:47:16 PM > [6/6] RUN python app/server.py:
Jun 19 03:47:16 PM ------
Jun 19 03:47:16 PM ------
Jun 19 03:47:16 PM error: exit status 1
Jun 19 03:47:16 PM error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c python app/server.py]: buildkit-runc did not terminate successfully
Here is the updated requirements file:
aiofiles==0.4.0
aiohttp==3.5.4
asyncio==3.4.3
fastai==1.0.60
numpy==1.15.4
Pillow==5.4.1
python-multipart==0.0.5
starlette==0.12.0
uvicorn==0.7.1
torch==1.4.0
torchvision==0.5.0
Here is the !pip list output from Gradient/Jupyter:
asn1crypto 0.24.0
attrs 18.2.0
backcall 0.1.0
beautifulsoup4 4.7.1
bleach 3.1.0
Bottleneck 1.2.1
certifi 2018.11.29
cffi 1.11.5
chardet 3.0.4
conda 4.8.1
conda-package-handling 1.6.0
cryptography 2.3.1
cycler 0.10.0
cymem 2.0.2
cytoolz 0.9.0.1
dataclasses 0.6
decorator 4.3.0
dill 0.2.8.2
entrypoints 0.3
fastai 1.0.60
fastprogress 0.2.2
idna 2.8
ipykernel 5.1.0
ipython 7.2.0
ipython-genutils 0.2.0
ipywidgets 7.4.2
jedi 0.13.2
Jinja2 2.10
jsonschema 3.0.0a3
jupyter 1.0.0
jupyter-client 5.2.4
jupyter-console 6.0.0
jupyter-core 4.4.0
kiwisolver 1.0.1
MarkupSafe 1.1.0
matplotlib 3.0.2
mistune 0.8.4
mkl-fft 1.0.10
mkl-random 1.0.2
msgpack 0.6.0
msgpack-numpy 0.4.3.2
murmurhash 1.0.0
nb-conda 2.2.1
nb-conda-kernels 2.2.0
nbconvert 5.3.1
nbformat 4.4.0
notebook 5.7.4
numexpr 2.6.9
numpy 1.15.4
nvidia-ml-py3 7.352.0
olefile 0.46
packaging 19.0
pandas 0.23.4
pandocfilters 1.4.2
parso 0.3.1
pexpect 4.6.0
pickleshare 0.7.5
Pillow 5.4.1
pip 18.1
plac 0.9.6
preshed 2.0.1
prometheus-client 0.5.0
prompt-toolkit 2.0.7
ptyprocess 0.6.0
pycosat 0.6.3
pycparser 2.19
Pygments 2.3.1
pyOpenSSL 18.0.0
pyparsing 2.3.1
pyrsistent 0.14.9
PySocks 1.6.8
python-dateutil 2.7.5
pytz 2018.9
PyYAML 3.13
pyzmq 17.1.2
qtconsole 4.4.3
regex 2018.1.10
requests 2.21.0
ruamel-yaml 0.15.87
scipy 1.2.0
Send2Trash 1.5.0
setuptools 40.6.3
six 1.12.0
soupsieve 1.7.1
spacy 2.0.18
terminado 0.8.1
testpath 0.4.2
thinc 6.12.1
toolz 0.9.0
torch 1.4.0
torchvision 0.5.0
tornado 5.1.1
tqdm 4.29.1
traitlets 4.3.2
typing 3.6.4
ujson 1.35
urllib3 1.24.1
wcwidth 0.1.7
webencodings 0.5.1
wheel 0.32.3
widgetsnbextension 3.4.2
wrapt 1.10.11
I can go into the virtual environment issue as well, but I’m wondering if this might not be the right thread for that?
Thanks!
Good afternoon cberkner I hope you are having a brilliant weekend.
I can report that your model with configuration above works though I used uvicorn== 0.11.5.
However using these to links I can get the bear classifier to work but not the jay bird classifier to work, the jaybird classifier gives the following error. _pickle.UnpicklingError: invalid load key, ‘<’.
export_file_url = ‘https://www.dropbox.com/s/6bgq8t6yextloqp/export2.pkl?raw=1’
#export_file_url = ‘https://www.dropbox.com/s/0skqq5a5odp2byh/export2.pkl?dl=0’
This error above nearly always means there is a problem with the sharelink, model or network the model is on. In this case the model works if I download it to the app.
I suggest you check the dropbox share link, use the google share link or if you have enough space use the model locally by saving it to the app directory.
Cheers mrfabulous1
1 Like
Thank you so much for taking all the time to troubleshoot this. I tried moving the .pkl file to Google and using a universal sharing link, but I got pretty much the same error:
Jun 23 12:59:13 PM #10 2.356 File “/usr/local/lib/python3.7/site-packages/torch/serialization.py”, line 529, in load
Jun 23 12:59:13 PM #10 2.356 File “/usr/local/lib/python3.7/site-packages/fastai/basic_train.py”, line 618, in load_learner
Jun 23 12:59:13 PM #10 2.356 state = torch.load(source, map_location=‘cpu’) if defaults.device == torch.device(‘cpu’) else torch.load(source)
Jun 23 12:59:13 PM error: exit status 1
Jun 23 12:59:13 PM #10 2.356 return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
Jun 23 12:59:13 PM #10 2.356 File “/usr/local/lib/python3.7/site-packages/torch/serialization.py”, line 692, in _legacy_load
Jun 23 12:59:13 PM #10 2.356 magic_number = pickle_module.load(f, **pickle_load_args)
Jun 23 12:59:13 PM #10 2.356 _pickle.UnpicklingError: invalid load key, ‘<’.
Jun 23 12:59:13 PM #10 ERROR: executor failed running [/bin/sh -c python app/server.py]: buildkit-runc did not terminate successfully
Jun 23 12:59:13 PM ------
Jun 23 12:59:13 PM > [6/6] RUN python app/server.py:
Jun 23 12:59:13 PM ------
Jun 23 12:59:13 PM error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c python app/server.py]: buildkit-runc did not terminate successfully
Since I have not yet been able to set up my virtual environment, and since Github won’t allow me to load a file larger than 25MB, I seem to still be stuck on this one.
Thanks again for all your help, even though it hasn’t solved my problem. I’m moving on to the next lesson and maybe when I figure out how to get my virtual environment set up I will come back to this.
Hi cberkner hope you have still had a wonderful day.
if you copy your model file to the app directory and change/amend the url line as below. it should work.
export_file_url = ‘’
export_file_name = ‘export2.pkl’
This is what I did.
I have created a working app and zipped it up. Its 300mb If you let me know where I can upload it to.
I can do that.
Have a Jolly day!
Cheers mrfabulous1
2 Likes
I did try to upload the .pkl file to the app directory in Github, but it won’t let me upload anything bigger than 25MB and the file is 80MB. I even tried upgrading my Github account but the 25MB limit remains.
I’m giving up for now. But I do think that Fast.ai should remove Render as a recommended platform for this purpose, don’t you?
3 Likes
I am stuck on the exact same error as you, Chris.
1 Like
Hi guys, any ideas what the problem here could be?
Here is a screenshot: Screen Shot 2020-06-25 at 00.55.52|690x212
Thanks.
Hi Adam_mehdi Hope all is well!
I would suggest you get the app working on your desktop first, then deploy it on render.com.
Cheers mrfabuous1
1 Like
Hi hodanajan Hope all is well!
I would suggest you get the app working on your desktop first, then deploy it on render.com.
Unfortunately the errors on render.com are not reflecting very well what is actually the problem in some cases.
Cheers mrfabuous1
1 Like
Thanks for the response!
I’m new to this and don’t quite understand what you mean by getting the app to work on the desktop; could you elaborate on that? How do I do that?
1 Like
I’ve just been struggling with this exact same problem as you two for the entire day (@LessW2020) : _pickle.UnpicklingError: invalid load key, ‘<’.
I was using Google Colab to train the model and strangely I think it was that I right-click downloaded the exported .pkl file from Colab and then re-uploaded again to Google Drive from my desktop. When I drag and dropped the file directly into the mounted Google Drive folder using the Colab interface and shared that file afterwards, the render build worked.
Perhaps the downloading/uploading to Google Drive in that way corrupted the pickle file which gave this error.
1 Like
Thanks Mike! Do you mind sharing your python file code? I tried Colab but I couldn’t figure out the proper filepaths. (And I assume you put your .pkl file in the Colab “sample_data” folder?)
Which python code do you need? From Colab or the render deployment? In Colab I just exported the learner without giving it a specific name using learn.export() which places the “export.pkl” file in the working directory (this was the default /content folder). Then I just drag n dropped the “export.pkl” file from there to the folder I wanted it to be in in my Google Drive that I had already mounted in the normal way as below (i.e. to the project folder I had already created in Google Drive):
from google.colab import drive
drive.mount('/content/drive')
1 Like
Thanks! Did you make any changes to the server.py file in the app folder?
In this case yes, as I’m using fastai-v2. So to get the render deployment working (in addition to the method mentioned above re drag n drop of export.pkl->Google Drive within Colab rather than download), I updated the files as follows.
requirements.txt - dependencies must match those in the Colab environment you trained the model in, as stated earlier in the thread by others. In my case:
aiofiles==0.4.0
aiohttp==3.5.4
asyncio==3.4.3
fastai==1.0.61
fastai2==0.0.17
fastcore==0.1.18
https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-linux_x86_64.whl
https://download.pytorch.org/whl/cu101/torchvision-0.6.1%2Bcu101-cp37-cp37m-linux_x86_64.whl
ipykernel==4.10.1
numpy==1.18.5
starlette==0.12.0
uvicorn==0.7.1
Pillow==7.0.0
python-multipart==0.0.5
server.py - a few small changes to imported libraries and prediction function (in addition to the Gdrive direct download URL, labels, “export.pkl” filename as necessary):
import aiohttp
import asyncio
import uvicorn
from fastai2 import *
from fastai2.vision.all import *
from io import BytesIO
from starlette.applications import Starlette
from starlette.middleware.cors import CORSMiddleware
from starlette.responses import HTMLResponse, JSONResponse
from starlette.staticfiles import StaticFiles
export_file_url = 'GOOGLE DRIVE URL'
export_file_name = 'export.pkl'
classes = ['YOURLABEL1', 'YOURLABEL2', 'ETC.']
path = Path(__file__).parent
app = Starlette()
app.add_middleware(CORSMiddleware, allow_origins=['*'], allow_headers=['X-Requested-With', 'Content-Type'])
app.mount('/static', StaticFiles(directory='app/static'))
async def download_file(url, dest):
if dest.exists(): return
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
data = await response.read()
with open(dest, 'wb') as f:
f.write(data)
async def setup_learner():
await download_file(export_file_url, path / export_file_name)
try:
print("File exists?:",os.path.exists(path/export_file_name))
learn = load_learner(path/export_file_name)
return learn
except RuntimeError as e:
if len(e.args) > 0 and 'CPU-only machine' in e.args[0]:
print(e)
message = "\n\nThis model was trained with an old version of fastai and will not work in a CPU environment.\n\nPlease update the fastai library in your training environment and export your model again.\n\nSee instructions for 'Returning to work' at https://course.fast.ai."
raise RuntimeError(message)
else:
raise
loop = asyncio.get_event_loop()
tasks = [asyncio.ensure_future(setup_learner())]
learn = loop.run_until_complete(asyncio.gather(*tasks))[0]
loop.close()
@app.route('/')
async def homepage(request):
html_file = path / 'view' / 'index.html'
return HTMLResponse(html_file.open().read())
@app.route('/analyze', methods=['POST'])
async def analyze(request):
img_data = await request.form()
img_bytes = await (img_data['file'].read())
pred = learn.predict(img_bytes)
return JSONResponse({'result': str(pred[0])})
if __name__ == '__main__':
if 'serve' in sys.argv:
uvicorn.run(app=app, host='0.0.0.0', port=5000, log_level="info")
Hope this helps.
2 Likes
Hi Adam_mehdi hope your well!
Basically you create a viritual environment on pc or mac
Copy the repository to that directory
Run pip install -r requirements.txt
See posts on this thread relating to ‘pip list’ and ediiting server.py
Once you resolve any library issues run
python app/server.py serve
This will return an webserver address that you enter in the browser.
You can then test your model in the browser.
Once you know it’s working deploying on render.com will be much easier.
Cheers mrfabulous1
1 Like