Jeremy's Harebrained install guide

SwiftAI
152

Thanks for catching the missing /lib64!

I corrected the LD_LIBRARY_PATH path and rerun python register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs --swift-toolchain ~/swift with the following output:

kernel.json:
{
  "argv": [
    "/home/user/anaconda3/bin/python",
    "/home/user/Documents/swift-jupyter/parent_kernel.py",
    "-f",
    "{connection_file}"
  ],
  "display_name": "Swift",
  "language": "swift",
  "env": {
    "PYTHONPATH": "/home/user/swift/usr/lib/python3.7/site-packages",
    "LD_LIBRARY_PATH": "/home/user/swift/usr/lib/swift/linux:/home/user/anaconda3/lib",
    "REPL_SWIFT_PATH": "/home/user/swift/usr/bin/repl_swift",
    "SWIFT_BUILD_PATH": "/home/user/swift/usr/bin/swift-build",
    "SWIFT_PACKAGE_PATH": "/home/user/swift/usr/bin/swift-package",
    "PYTHON_LIBRARY": "/home/user/anaconda3/lib/libpython3.7m.so"
  }
}

Registered kernel 'Swift' as 'swift'!

I guess the output looks correct?

However, I still get the same error when running swift in the shell and the kernel in jupyter notebook still behaves as a normal python kernel.

In addition I tried that:
Because you mentioned that you have a cuda-10.0 in /usr/local/ directory I created a symlink with this name there which links to the cuda-10.1 directory but this didn’t help.

153

Did you pick Swift as your kernel?

45%20AM
Screen Shot 2019-05-05 at 10.32.45 AM.png670×508 42.9 KB

154

Also, you’re now missing envs/swift/lib from the register.py output, which you had previously.

155

This is there, or am I misunderstanding it?

I switched to the swift kernel in the notebook. However, mine is written with lower-case letters, i.e., “swift”. Should it be named “Swift” like in your screenshot?
(I was asking this myself after running register.py which says Registered kernel 'Swift' as 'swift'!.)

Under /anaconda3/envs I only have a directory named “swift” (and no “Swift”).

My “swift” conda env I created with conda create --name swift prior to the installation steps.

Maybe, there is a way to get the the output of the actions carried of the register.py script?
Is there a way to look for the changes that should have been carried by the register.py script?

156

Your most recent register.py output is missing the /envs/swift

You might not have activated the swift conda environment where you ran register.py. You need to:

  1. source activate swift
  2. Run register.py etc.
  3. Start jupyter notebook
  4. All in the same shell in the same go
1 Like
157

I tried everything again:

1.) conda activate swift

2.) Set LD_LIBRARY_PATH:

$ echo $LD_LIBRARY_PATH
/usr/local/cuda/lib64:/home/user/anaconda3/envs/swift/lib

3.) Run register.py:

$ python register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs   --swift-toolchain ~/swift
kernel.json:
{
  "argv": [
    "/home/user/anaconda3/envs/swift/bin/python",
    "/home/user/Documents/swift-jupyter/parent_kernel.py",
    "-f",
    "{connection_file}"
  ],
  "display_name": "Swift",
  "language": "swift",
  "env": {
    "PYTHONPATH": "/home/user/swift/usr/lib/python3.7/site-packages",
    "LD_LIBRARY_PATH": "/home/user/swift/usr/lib/swift/linux:/home/user/anaconda3/envs/swift/lib",
    "REPL_SWIFT_PATH": "/home/user/swift/usr/bin/repl_swift",
    "SWIFT_BUILD_PATH": "/home/user/swift/usr/bin/swift-build",
    "SWIFT_PACKAGE_PATH": "/home/user/swift/usr/bin/swift-package",
    "PYTHON_LIBRARY": "/home/user/anaconda3/envs/swift/lib/libpython3.7m.so"
  }
}

Registered kernel 'Swift' as 'swift'!

4.) Run jupyter notebook in the same bash window.

5.) Start fastai_docs notebook and run the first cells:

image
image.png1566×1212 171 KB

(Jupyter in the command line does not throw errors.)

6.) When I run swift in the shell I get this new error:

$ swift
swift: /home/user/anaconda3/envs/swift/lib/libuuid.so.1: no version information available (required by swift)
/home/user/swift/usr/bin/lldb: /home/user/anaconda3/envs/swift/lib/libuuid.so.1: no version information available (required by /home/user/swift/usr/bin/../lib/liblldb.so.7)
error: failed to stop process at REPL breakpoint

(There is also a thread about a similar output in the notebooks: [minor] "No version information available" warning when importing modules)

To run everything in the same shell was essential (and this is something I am not used to with other python kernels I regularly use). I guess the problem was that when you are writing the paths to .bashrc you have to restart the bash to have it there too.

However, with this no version information available I didn’t really found a source that helped me.

I am making small steps, but those are still steps. :wink:

158

It seems to me that you may not be actually running a Swift kernel. The kernel name shows lower case ‘swift’ in your screenshot, but it should be registered with upper case based on the register.py output. The fact that it doesn’t recognize the magic or the comment is suspicious. You could try running some Python to see if it’s actually a Python kernel.

I have experience with the error you see in the Jupyter notebook. I saw it when I updated S4TF. I believe I resolved it by updating swift-jupyter, running register.py, then starting jupyter. It actually may or may not matter if it is in the same shell but that is what I did.

What version S4TF and swift-jupyter are you running? I synched up as of right before Lesson 14. I can’t vouch for anything that doesn’t match that.

What you experience in Jupyter and what you experience when you run swift in the command line aren’t necessarily completely related. So you are potentially describing two totally different issues. I think you probably want to focus on Jupyter.

As far as Jupyter goes, what register.py does is set up the context for the Jupyter kernel named ‘Swift’ (in the UI). You can look at the source.

I would avoid adding the anaconda one in here. And definitely don’t do this in your bash init. But I don’t think it is necessarily causing your issue.

159

Thanks for your help.

My (lower-case) swift kernel in jupyter behaves like a normal python kernel and I can run python code with it.

I used swift-tensorflow-RELEASE-0.3.1-cuda10.0-cudnn7-ubuntu18.04.tar.gz and the latest swift-jupyter repo.

I think so too, that the problem in jupyter is not related to the one in the shell.
It looks like the (upper-case) swift env is not installed properly (even when the script is not showing an error).

However, I don’t know what else I could try.
I guess I will switch to colab or use docker until the next release.

160

Yeah, not sure what the issue is.

I know I encountered the same problem you are having in the notebook, but it was pretty easy to resolve by updating and running register.py again.

161

After removing the conda “swift” env. and the swift folder I see now a “Swift” environment in the jupyter kernels.

Maybe the two “swift” and “Swift” env. where hindering each other?

From the install script in the first post it seems like the conda “base” env. is used?

I’ll try it again and I am happy for every tip! :slight_smile:

162

Hi, I had the same issue with libuuid.so.1 and i used this which solved my issue.

python register.py --sys-prefix --swift-toolchain ~/Applications/swift-tensorflow --swift-python-library ~/miniconda/envs/s4tf/lib/libpython3.6m.so

put your path to libpython3.6m.so

1 Like
163

For those of you creating new Google Compute Engine instances, what settings are you using? The default fastai GCP instructions install Debian I believe. Are you using the Ubuntu 18.04 LTS Image and the same fastai hardware settings recommended in the GCP guide? Also if you already had a Google VM for fastai, are you migrating that onto the new Ubuntu machine or using the Ubuntu machine exclusively for Swift?

164

Just following up for anyone else wondering, I set up a new VM instance in my existing fastai project with the commands outlined in step 3 of the GCP Start Guide. I set IMAGE_FAMILY=“ubuntu-1804-lts”, renamed INSTANCE_NAME and set --image-project=ubuntu-os-cloud. Otherwise, I followed the instructions exactly for creating a new instance. Then installed everything per Jeremy’s gist and was successfully able to start the first notebook.

6 Likes
165

I’m trying to setup Swift (v0.4 of S4TF), I’ve managed to get the notebooks running but I am running into a strange error in 00_load_data.ipynb
When I try to define the new protocol:

protocol ConvertibleFromByte: TensorFlowScalar {
    init(_ d:UInt8)
}

I get this error:

error: Couldn't lookup symbols:
  protocol descriptor for TensorFlow.TensorFlowScalar
  protocol descriptor for TensorFlow.TensorFlowScalar
  protocol descriptor for TensorFlow.TensorFlowScalar
  protocol descriptor for TensorFlow.TensorFlowScalar

TensorFlow.TensorFlowScalar is defined but the error message suggests that doesn’t have a protocol descriptor?

166

Hmmm… it seems that I had not installed cuda 10 into my system properly. Installing cuda10 by more carefully following jeremy’s minimal installation subset has helped and I’m on to the next error :slight_smile:

168

I got this

image
image.jpg1038×696 215 KB

What’s the problem?
It’s too hard to install

169

I managed to get s4tf working on my desktop. I don’t have a step by step guide, but here are a few tips that might help you :slight_smile:

I was mostly following the script that jeremy posted at the top of this thread, but a few things caught me up, probably because I was not running on a clean dev environment. One thing I’ve skipped here is setting up Cuda, don’t forget to do that :slight_smile:

swift-jupyter is really important, make sure you get this running. Note that it requires Python 3.6 (not 3.7 which is current anaconda python 3), so check your python version.
python --version
One quick way is to create a new anaconda environment with the command:
conda create -n swift-tensorflow python=3.6
Don’t forget to install tensorflow and any other libraries you want into the new environment.
conda install matplotlib scikit-learn tensorflow-gpu jupyter numpy

When you finally setup swift-jupyter (using the command python register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs --swift-toolchain ~/swift) it prints a bunch of environment variables, make sure to capture these and you can put them into your ~/.bashrc. For me the variables were:

export PYTHONPATH=/home/john/swift/usr/lib/python3.6/site-packages
export LD_LIBRARY_PATH=/home/john/swift/usr/lib/swift/linux:/home/john/anaconda3/envs/swift-tensorflow/lib:$LD_LIBRARY_PATH
export REPL_SWIFT_PATH=/home/john/swift/usr/bin/repl_swift
export SWIFT_BUILD_PATH=/home/john/swift/usr/bin/swift-build
export SWIFT_PACKAGE_PATH=/home/john/swift/usr/bin/swift-package
export PYTHON_LIBRARY=/home/john/anaconda3/envs/swift-tensorflow/lib/libpython3.6m.so

If you use these, don’t forget to change the paths to your home folder and your anaconda environment.
After doing this, I was able to compile the s4tf example swift -O inference.swift in the s4tf usage instructions. This should let you compile swift code from the command line as well as using the jupyter notebooks locally.

Also, don’t forget to replace any reference to fastai_docs to course-v3 as the notebooks have been moved.

170

Hard to say what went wrong as most of the output is missing. From what you posted it looks like anaconda may not have been installed (perhaps the download failed) but the script continued working, cloned a few repo’s but then since anaconda hadn’t installed, jupyter wasn’t available at the end.
The error “source: not found” worries me abit, what shell/OS are you using?

171

Hi,
Great to hear, I have the same set up as yours.
can you please explain the steps you took ?
I didnt find any “conda create”

Thanks!
Nadav

172

Anyone working on setting up S4TF on macOS? Seems trivial inside Xcode and horribly painful outside.