@ashtonsix Thank you! After creating a brand new server it works great.
@bluesky314 Both untar_data and ImageDataBunch work in my new Salamander server.
Others may be interested in the versions of components, so this is what’s showing for me (I got a 1xK80 server):
python version : 3.7.0
fastai version : 1.0.11
torch version : 1.0.0.dev20181020
nvidia driver : 396.54
torch cuda ver : 9.2.148
torch cuda is : available
torch cudnn ver : 7104
torch cudnn is : enabled
Please anyone let me know if it’s inappropriate to post these details and I’ll be happy to remove them.
@Kaushikjais Soon! I’m adding a way to add promotional credits to your account & it’ll be available either later today or tomorrow. rn my main focus is service reliability as some users have reported issues with servers getting stuck while starting / stopping
Hello @ashtonsix .my notebook is not connecting to the kernel successfully .keeps disconnecting and trying to reconnect. Please advise i urgently need to so some work
@Aya You mean, you’re able to open Jupyter Notebook but once open it can disconnect & you never see any results? If so, that’s an issue with Jupyter Notebook itself rather than Salamander. It can be caused by bad network conditions.
From the Jupyter Notebook team:
Anything already running in the notebook will keep running, and the kernel it started for that will stay running - so it won’t lose your variables. However, any output produced while the notebook isn’t open in a browser tab is lost.
See this thread for more details. For now, the only thing I can think of would be trying a different wifi / network or maybe plugging yourself into ethernet.
Hi @ashtonsix! Thanks for the feedback. I excluded the bas network option because YouTube was running fine .but you are probably Correct since it worked when i moved to another place. Thanks alot !
I am finding that training times on Salamander are much slower than the training times in the lecture. One epoch that takes 16 seconds in the lecture is taking me 3:55. This is on a U-net (frozen) but I find it true on CNNs as well. Is this expected? Thanks.
@Mark_F in the lectures I often use V100 GPUs which are the fastest available (but also very expensive!) You shouldn’t generally expect to see the same performance on other machines.
SSH keys: I’ve uploaded some public keys via the web interface. I can see they are added to /home/ubuntu/.ssh/authorized_keys whenever I launch a new instance. Now I’d like to know how I can remove them so they aren’t added automatically on launch?
If your server isn’t working, try opening a terminal and conda install -c pytorch -c fastai pytorch fastai. Then restart it. If that doesn’t work, try conda update --all.
I made a new account to try Salamander today - worked with no issue after updating server with the conda update.
Came back to do more work later…hit the 500: Internal Server Error many have hit here.
After looping around a bit found this thread, did the conda update --all.
Now I can open my notebook but I never get a working kernel - it just stays in “connecting to kernel” mode meaning no working notebook…I then shutdown the whole server, restarted and still get the exact same issue. It’s this:
Thanks, I destroyed the old server since I wasn’t clear what was going wrong and made a new one that worked great today.
btw - the Salamander CPU’s are waaay faster than my previous platform. File uploads/unzips, etc. are about 20x faster which is really nice.
Thanks for the clarification, at least now I know it was a one time deal and not something I’ll hit going forward!
