How to Set Up Jupyter Notebooks for S4TF on Docker

regrettable-username · March 15, 2019, 7:53am

Hey, everyone!

I just wanted to briefly share my excitement for fastai and S4TF. It goes without saying that I’m already a huge fan of fastai–the teaching philosophy really resonates with me. I was already extremely excited to be attending the 2019 Deep Learning Pt. II course, but hearing the news at TF dev summit had me cheering! I was at WWDC when the open sourcing of Swift was announced (equal excitement). I truly think Swift is an amazing language. And that is coming from an objective place. I regularly have to go back and forth between Swift, Obj-C, C and C++ for work, and I have to say, I feel so much more productive with Swift. I’ve had colleagues independently say the same thing. I’m super optimistic to say the least!

Anyways, I thought I’d share the medium post I just made about getting going with swift-jupyter on Docker (feedback is more than welcome of course). It was the cleanest way I found to try stuff out on my compute rig. I’m also working on some other guides on getting going via other methods including Xcode. I’m also looking at other ways of contributing back but need to reel myself back into PyTorch and fastai stuff a bit. I still have a lot of cool stuff to learn w.r.t. deep learning.

Cheers!

radek · March 15, 2019, 9:10am

This is really cool! Thx for sharing. Really nice, detailed write up.

I’m like on my 0.75 day of learning Swift and so far enjoying the experience a lot

neuradai · March 21, 2019, 3:50am

I just wrote up my experience setting up S4TF + Jupyter on Ubuntu 18.04 without Docker, posted on Medium. Then saw this thread and thought I’d share it here.

cwerner · March 22, 2019, 7:03pm

Eager to get into it, too.

I wonder (and maybe I missed this): is it possible to play around with swift4tf on a MacBook Pro with Xcode but obviously no Nvidia GPU?
Can you also do the python imports in this setup?

Or should I go via jupyter?

Sorry for the noob question,
C

saeta · March 22, 2019, 7:29pm

There are a lot of ways to get started with Swift for TensorFlow, and no NVidia GPUs required! Check out the README which includes instructions for how to get started. Google Colaboratory lets you get started right in your web browser, which is super fast. But if you’d like to run things locally, check out the local installation instructions right below.

All the best,
-Brennan

cwerner · March 22, 2019, 8:11pm

Thanks! Not sure how I missed that

saeta · March 22, 2019, 8:52pm

No worries!

dan-zheng · March 22, 2019, 9:37pm

Hi Walt,

Thanks for documenting your setup experience on Medium! I noticed this blurb at the bottom:

Could you please share more info about your errors? We’re definitely interested in fixing them.

cwerner · March 22, 2019, 11:11pm

@dan-zheng hope you don’t mind me jumping in with a question re python interop.

Is there a way to use a specific python version on the system (I.e. installed via pyenv)?

saeta · March 22, 2019, 11:52pm

I believe you can use:

PythonLibrary.useVersion(2, 7)

to set a specific Python version. Is that what you’re looking for?

dan-zheng · March 23, 2019, 12:15am

Yes - I’m sorry that this is currently undocumented.

If you have a path to the libpythonX.x.so shared library for the desired Python environment you want to use, set the PYTHON_LIBRARY environment variable to it.

# Example.
export PYTHON_LIBRARY="~/anaconda3/lib/libpython3.7m.so"
# Then, try running your Swift program again.

Here’s a Gist explaining this and other Python details. I’ll document this more formally when I get a chance.

neuradai · March 23, 2019, 3:38am

This Gist got me on the right track. It took some experimentation to sort everything out, but I finally got there…at least for Swift-Jupyter…

These are the steps for my conda env setup and registration of the Swift-Jupyter kernel.

$ conda create -n s4tf python==3.6
$ conda activate s4tf
(s4tf) $ conda install jupyter numpy matplotlib
(s4tf) $ python register.py --sys-prefix --swift-toolchain ~/Applications/swift-tensorflow --swift-python-library ~/miniconda3/envs/s4tf/lib/libpython3.6m.so

After these steps, I was able to run the tutorial Jupyter notebooks without any errors.

Out of curiosity, I next tried to set the PYTHON_LIBRARY environment variable, to then run the Swift REPL and play around with Python interop… This was the result.

I’m pretty content to use Swift-Jupyter for now, but would appreciate any suggestions you might have as to how this might be remedied for future use of the Swift REPL.

dan-zheng · March 23, 2019, 4:50am

Is it the case that setting PYTHON_LIBRARY fixes Jupyter but not the REPL? If so, that’s a bit unexpected to me.

Could you please try export PYTHON_LOADER_LOGGING=1 as well?

You should be able to see some debug output like this, that should help illuminate things.

neuradai · March 23, 2019, 5:09am

Exporting PYTHON_LOADER_LOGGING=1 didn’t seem to change anything. No additional debug output was printed. It seems like the REPL isn’t seeing my environment variables for some reason.

cwerner · March 23, 2019, 11:48am

Thanks.

I will check it out!

Borz · March 29, 2019, 5:18pm

Has anyone successfully setup Jupyter for S4TF on macOS without Docker? Looks like the main challenge is to install a Swift toolchain with LLDB Python3 support – and for non Ubuntu 18.04 systems that requires building from sources.

See:

swift-jupyter Installation Instructions
- note on building toolchain for non-Ubuntu systems
- section on building toolchain w/ LLDB Python3 support

I’ll update once I get it working.

Currently hitting the error:

In Getting Sources for Swift and Related Projects (HTTPS):

git clone https://github.com/apple/swift.git -b tensorflow
./swift/utils/update-checkout --clone --scheme tensorflow

That last line causes:

Traceback (most recent call last):
  File "./swift/utils/update-checkout", line 5, in <module>
    import update_checkout
  File "/Users/WayNoxchi/Developer/swift-source/swift/utils/update_checkout/__init__.py", line 2, in <module>
    from update_checkout import main
ImportError: cannot import name 'main'

Not something I’ve seen before. Doesn’t exactly look like a similar pip error other’s experienced.

So, in swift-source/swift/utils/update_checkout/ , __init__.py is trying to do:

from update_checkout import main

__all__ = ["main"]

And then it can’t find the name “main”. __init__.py is inside …/update_checkout/ so I’m not sure what/how it’s importing … but since this is Python “main” should be referencing the original .py file running … so if it can’t find that something either very serious or very silly went wrong.

update to that:

From the Swift forums: Problem in building swift for tensorflow;

Looks like the ghost of Python 2 is still out there… you can follow Timothy Vincent’s post exactly. Rerunning in a Python 2.7 environment got everything to start working. I submitted a PR for the ReadMe.

issue 2: currently held up when running:

SWIFT_PACKAGE=tensorflow_linux,no_test ./swift/utils/build-toolchain local.swift

a modification in google/swift-jupyter:(optional) Building toolchain with LLDB Python3 support to replace running utils/build-script in apple/swift:Building Swift with TensorFlow support.

Error:

./utils/build-script: note: Using toolchain default
+ mkdir -p /Users/WayNoxchi/Developer/swift-source/build/buildbot_linux
./utils/build-script: fatal error: can't find source directory for libicu (tried /Users/WayNoxchi/Developer/swift-source/icu)
./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

I guess I have to find/install libicu then. Is it because I may/not have the Python 3 deveopment headers? The google/swift-jupyter ReadMe only gives installation instructions for Ubuntu 18.04.

update:

Following along installing libicu-dev on mac, looks like I already have icu (icu4c)… so do I point clang / swift there (and how)? Or is there something else missing…