@sujithjose5 it takes a while for a newly launched server to “warm up”, since it has to transfer all its data from s3 the first time it is accessed. After you’ve trained a single epoch, all the data will have been transferred, and you should see things run fast again.
1 Like
Here’s an update I just sent all customers via email (although it’s currently stuck in Postmark’s queue, so might be a while before folks receive it):
I’m sure those of you that have been using Salamander recently have noticed that there’s been a severe shortage of the default type of server that we provide, the “Accelerated Computing” instance type. Unfortunately, our provider (Amazon Web Services) had a shortage of this server type, which we weren’t aware of until it started impacting Salamander customers.
We know that this has resulted in a lot of frustration and wasted time for many of you, as you have been unable to access your servers during this time. We are very sorry that this happened.
The good news is that there is now a faster and cheaper instance type available: the “g3s” instance, part of the “Workstation Graphics” selection. It’s what we’ve used for developing and teaching much of the fast.ai course. We’ve been checking this instance type every day, and it’s been available every time we’ve checked. It uses a GPU that’s about twice as fast as the “Accelerated Computing” instance, and is far cheaper (the Accelerated Computing instance is generally around 40% more expensive). The only downsides are that the g3s have less RAM and less GPU memory.
We have added the g3s option to everyone’s account. However we haven’t changed your current servers; to do so, click the “Modify” button next to any stopped server, and click the ‘g3s’ button, then click ‘Replace Server Hardware’.
In other news, we’ve disabled signups of new accounts for now, since we want to focus on serving existing customers effectively, rather than bringing on more customers.
1 Like
Hi everybody,
I’m having a problem with salamander again, my server starts, but Jupyter Notebook won’t. There might be something super basic that I’m missing, but I’m not able to figure it out…
After the ‘no server available’ issues described above, everything was running fine, using the Accelerated Computing v2. However, while running a notebook yesterday, I got an error message related to the memory use. I tried to restart my instance, but that didn’t work: the server starts, but Jupyter Notebooks won’t… A new tab in my browser opens, but it says ERR_CONNECTION_REFUSED.
Since then I
- tried different browsers (chrome and edge)
- tried different WiFi networks and providers
- changed to different server hardware (e.g. g3s)
- increased my storage space (from 75GB SSD to 150GB SSD)
- switched to Jupyter Lab, but that’s also not working
After that, I made a new, second server (on the same salamander account) with the same specs. That one runs fine, and so is Jupyter Notebook. However, all my work and personal notebooks are on the first server. I have backups of my notebooks, but not of my saved models, so I would still like to regain access to my first server
I searched the fast.ai forum to see if anybody else had the same problems, but I couldn’t find anything. Maybe it has to do with the SSH keys? But that seems a bit weird to me, as I didn’t change anything and the second server is working fine…
Any advise would be highly appreciated…
Cheers,
Jeroen
connect to your first server via SSH and try starting Jupyter Notebook from there
I could not open my Jupyter with the message below:
“We started your server, but Jupyter isn’t running yet (it’s taking longer than usual)”
Does anybody know how to fix it?
I have waited more than a hour and still cannot start my Jupyter although the machine seems to be running.
You’ll have to ssh into it to figure out why jupyter isn’t starting.
@jeremy Thank you for your suggestion.
I am using PuTTY but could not get into it, because of “Network error: Connection timed out.”
I am very new with it, so perhaps miss something…
Following this instruciton, double checked IP and use port 22.
Anything else do I have to configure?
this can happen on Ubuntu when you try to SSH in within a couple minutes of your server starting. wait for a little bit and try again (if you waited over 15 minutes and still see this error then it’s something else and i don’t know how to help)
Hi everyone,
In an attempt to create a language model for Ukrainian, I try to upload to my Salamander folder a wikipedia dump c.a. 1.5 GB volume. The upload process stops at 1.0 GB. Does it mean there exists some upload limit?
first I want to say thanks Ashton and the fastai team for their amazing work!
Second, who should i talk to about my account being reset by mistake? I have receipts from yesterday and today for purchased credits, and less than an hour after an email saying my account is to be deleted tomorrow because it could not process my card I saw my account was reset with a message saying there was no response from me for 7 days.
This is clearly false. I hope there is some error somewhere, though there is(nearly) no data I had on the server that cannot be recovered elsewhere, i’ve been training something for many days now and it would be a shame to lose any of it. I hope this is able to be rectified, thanks for the great services.
Hi,
I’m having an issue with my Salamander server not stopping, it has been stuck at this stage for about 4 hours now and has thus far cost me an entire coffee.
It occasionally comes up with a message saying that this is happening because I tried downloading a large data set… I did download a kaggle dataset before the server stopped, but that was to my computer’s hard drive and I never uploaded it to my Salamander server, so that doesn’t quite add up to me. Is there something wrong here? Or do I just need to wait it out and eat the costs?
Thanks for all your work, Salamander and fast.ai have been great to work with! (up until now)
That does seem odd @Cameron - it does take quite a while to save your work when you have a lot of data on it, but that doesn’t seem to be the case here. Can you send me a PM with your username?
Update: resolved.
Hello
Just letting you guys know that my server page is stuck on “Offline - moving storage” for about 2 hours now. i have about 700MB of data, so that shouldn’t be the reason.
Cheers
AP
Hi,
I tried Salamander to begin the course last October but there were some issues at the time so I switched to GCP instead.
I was auto-charged 20$ in November so I deleted my payment information.
To my surprise, today I was charged 20$ again for storage.
On reading the FAQ I realised it was my mistake as I should have deleted my account completely, instead.
Now I have approx 20 dollars credit, which I won’t be using because I am comfortable using GCP.
Are there refunds possible? Or should I just delete account now to avoid another 20$ charge?
Best would be to use the credits, now you have them.
Note that you don’t have to delete your account - you can just destroy the instance you have.
If you really need the refund, then delete your account and then PM me your login email.
1 Like
Hi all - will Salamander be opening to new customers soon, or is this a long term block for new customers?
I am working with two clients that want to setup their own servers for training so wanted to be able to address this issue with them.
Thanks!
Less
It’ll be until we have time to work on the platform - so at least a few more months. Sorry!
1 Like
Hi, I’ve been away for almost 2 months. I just added $20 to my Salamander account have $21.84 balance - keen to get back to work! I am getting tossed from the server, and am finding it difficult to get a new one. Should I persist, or ask for a refund and move to another service?
Cheers,
–Peter