How to use/install a Swift library in Collab

SwiftAI
1

Is it possible to use/install a Swift library (e.g. Just) on Collab?

In python it would have been as easy as:

!pip install git+https://github.com/org/repo.git
2 Likes
2

I tried the following but it’s not working

%install '.package(url: \"https://github.com/JustHTTP/Just\", from: \"0.7.1\")' Just

I got those errors:

error: <Cell 45>:1:10: error: single-quoted string literal found, use '"'
%install '.package(url: \"https://github.com/JustHTTP/Just\", from: \"0.7.1\")' Just
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         ".package(url: \"https://github.com/JustHTTP/Just\", from: \"0.7.1\")"

error: <Cell 45>:1:9: error: consecutive statements on a line must be separated by ';'
%install '.package(url: \"https://github.com/JustHTTP/Just\", from: \"0.7.1\")' Just
        ^
        ;

error: <Cell 45>:1:80: error: consecutive statements on a line must be separated by ';'
%install '.package(url: \"https://github.com/JustHTTP/Just\", from: \"0.7.1\")' Just
                                                                               ^
3

Quoting myself from another topic:

Sorry for the inconvenience!

4

thanks @dan-zheng I’ve a mac, i.e. no gpu, it’s going to be slow even if I get it to work locally.

5

You can follow the instructions under “getting newer swift builds” here: https://github.com/tensorflow/swift/blob/master/Usage.md#colaboratory to try out v0.3 before it actually gets released to Colab.

6

Brilliant, thanks @marcrasi, now I’m getting this error with the able install

Install Error: Cannot install packages because SWIFT_IMPORT_SEARCH_PATH is not specified.

What should be the value for SWIFT_IMPORT_SEARCH_PATH env variable?

7

I’ve seen this error before, and I think is has something to do with the new version getting somehow half-installed to Colab. Not sure exactly what triggers it, but it seems to have something to do with having Colab tabs open while you run the installation notebook. These steps always fix it for me:

  1. Close all Colab tabs.
  2. Open only the installation notebook.
  3. Runtime > Reset all runtimes
  4. Run the installation notebook and wait for it to completely succeed
  5. Finally, open a Swift notebook
3 Likes
8

Yeah indeed I have another tab open, I messed up until I get rid of the error, I think it ends to the steps you laid down.

9

I am unable to run the Lesson 13 notebook 00_load_data.ipynb in Google Colab. Can anyone help?

Here is what I did:

First, in Firefox (Windows 10 64-bit), I ran install_latest_swift.ipynb, with the result

The newly installed Swift build is:
Swift version 5.0-dev (LLVM dcb9eb74a7, Clang 95cdf7c9af, Swift dc31c3fcd2)

Then I obtained a fresh-out-of-the-oven copy of the notebook, directly from the fastai/fasta_docs/dev_swift/ folder in the current fastai master repo. Here is how: from the menu in the install_latest_swift.ipynb notebook that you just opened, choose File, Upload notebook..., choose the GITHUB tab from the popup menu, on the first line type fastai/fastai_docs and hit Return to specify the repo (the branch shows as MASTER), scroll way down through the list of files to find the notebook at dev_swift/00_load_data.ipynb, then (finally!) click the arrowed box at the right of the selected file name, which opens the notebook in Colab, in a new tab.

Selecting Runtime from the menu across the top of the notebook, then Change runtime type from the resulting pull-down menu shows that the Hardware accelerator state is None. An attempt to run the first cell in that notebook first pops up a window with a warning: uncheck the Reset all runtimes box and click Run anyway. Then it throws this error:

Kernel is in a bad state. Try restarting the kernel.
Exception in `_process_installs`:
[Errno 17] File exists: '/content/swift-install' -&gt; '/tmp/tmpitzk7hqp/swift-install'

Selecting Runtime, Change runtime type from the menu, setting Hardware accelerator to GPU, and again attempting to run the first cell throws this error on the first line:

error: <Cell 2>:1:18: error: consecutive statements on a line must be separated by ';' 
%install-location $cwd/swift-install

Curiously, there are two different failure modes, one when Hardware accelerator state is None, one when Hardware accelerator state is GPU!

What have I done wrong? Does anyone know how to make this work?

10

Yes, there appear to be a few different problems.

  • The first one is that the magic directive %install-location is not working in Colab for some reason. As a workaround, you can remove it from the first cell in the notebook for now.

  • The second one is that changing the runtime type seems to place the notebook in a different environment, where the %install directives are not available. I confirmed this by listing the contents of /swift/swift-jupyter from a Swift GPU notebook and from a regular Python one - they were different after enabling the GPU. This might be resolved when the 0.3 version is deployed, perhaps @marcrasi can confirm if that’s the case. I didn’t find any workaround for this problem, so I/m afraid you’ll have to use the CPU runtime until this is resolved.

To go back to testing, please follow the procedure described by Marc above: close all your notebooks, open the installation one and run it. Then open 00_load_data.ipynb and remove the first line (the %install-location one) from the first cell.

1 Like
11

@jcatanza Actually, there seems to be a workaround:

  • Go to the Runtime | Manage sessions menu item in Colab and terminate all your sessions.
  • Open the installation notebook and run it.
  • Before closing it, enable GPU acceleration and re-run the installation cell. Repeat this step for all the runtime types you plan to work on.
  • Open the lesson notebook and remove the first line as indicated.

@marcrasi hostname shows a different name after GPU acceleration is enabled. I enabled GPU acceleration (for the Python installation notebook) and re-run the installation process. I confirmed the hostname matches the one I was seeing in the Swift notebook, but then I get this error when running any cell in the GPU-enabled Swift notebook: Install Error: Cannot install packages because SWIFT_IMPORT_SEARCH_PATH is not specified.

I’m not sure how the Colab environment differs from a regular Jupyter one, but will try to investigate somehow. I’m guessing differences in the jupyter kernel startup sequence?

2 Likes
12

Thanks @pcuenq

OK so I followed your instructions.

I loaded the install_latest_swift notebook, reset all runtimes, executed the notebook, changed the runtime to GPU, re-executed the notebook.

Then I uploaded the 00_load_data notebook directly from the GitHub repo. I changed its runtime to GPU, deleted the first line, and ran the first cell. A window popped up and asked if I was sure I wanted to run it (it had a checked box that said reset all runtimes, and I left this box checked). I hit “run anyway” and this time I got a different error:

Installing packages:
.package(url: “https://github.com/mxcl/Path.swift”, from: “0.16.1”)
Path
.package(url: “https://github.com/JustHTTP/Just”, from: “0.7.1”)
Just
.package(url: “https://github.com/latenitesoft/NotebookExport”, from: “0.5.0”)
NotebookExport
With SwiftPM flags: []
Working in: /tmp/tmp0a5fymwl/swift-install
/swift/toolchain/usr/bin/swift-build: error while loading shared libraries: libBlocksRuntime.so.0: cannot open shared object file: No such file or directory

Install Error: swift-build returned nonzero exit code 127.

This time, when I repeated the process without the GPU, I got the same error.

Any idea how to work around this? Thanks!

Joseph

13

This is the problem :slight_smile: This box deletes your VM, including the newer toolchain that you installed.

Tomorrow (Tuesday April 30) around noon Pacific time, most of these steps should no longer be necessary because most of the upgrades and fixes will be released! (You’ll still need to delete the %install-location because the fix for that will take a few more days to release.)

2 Likes
14

@marcrasi I’ve just tried running %install on a new blank notebook and I got same error as above

15

Sorry! There was an unrelated bug in Colab this morning that forced them to roll back to the Friday release.

So you will have to use the “install latest notebook” for at least another day to use the %install feature.

A new release should happen tomorrow, unless anything else goes wrong.

2 Likes
16

@marcrasi OK I’m checking the “reset all runtimes” box did the trick. I was able to execute the notebook down to the 3rd from the last cell.

For some reason, the notebook is not visible in the path…

The second from the last cell

notebookToScript(fname: Path.cwd/"00_load_data.ipynb")

threw this error:

Can't read the content of /content/00_load_data.ipynb

Following up, I noticed that the command in the 5th cell that lists what is in the current directory

print("/bin/ls".shell("-lh"))

tells us that the current directory contains only a folder named sample_data. There are no .ipynb notebooks,

total 4.0K
drwxr-xr-x 1 root root 4.0K Apr  4 20:20 sample_data

So the question is: why am I not seeing the .ipynb notebooks, and how can I get into the directory with the notebooks?

17

Take a look at the last paragraph (“There is one additional problem…”) in this post here: S4TF in colab (error) - #4 by marcrasi. It should make this error go away, and should also fix similar problems in subsequent notebooks.

Even though that will fix the error, notebookToScript won’t really work like you would expect in Colab. It exports the notebook that is on the disk into a package that can be imported into subsequent notebooks. But when you’re working in Colab, the notebook on disk is not the same as the notebook that you have opened. As long as you’re not trying to import your modifications into subsequent notebooks, this won’t be a problem.

1 Like
18

@marcrasi still not able to install external libraries in colab with, I tried with the blank swift kernel then with the setup instructions https://github.com/tensorflow/swift/blob/master/Usage.md#colaboratory

> %install '.package(url: \"https://github.com/JustHTTP/Just\", from: \"0.7.1\")' Just
Installing packages:
	.package(url: \"https://github.com/JustHTTP/Just\", from: \"0.7.1\")
		Just
With SwiftPM flags: []
Working in: /tmp/tmpw6hpbp41/swift-install
/tmp/tmpw6hpbp41/swift-install/package: error: manifest parse error(s):
/tmp/tmpw6hpbp41/swift-install/package/Package.swift:11:35: error: unterminated string literal
    dependencies: [.package(url: \"https://github.com/JustHTTP/Just\", from: \"0.7.1\"),
                                  ^
/tmp/tmpw6hpbp41/swift-install/package/Package.swift:21:1: error: expected ')' in expression list

^
/tmp/tmpw6hpbp41/swift-install/package/Package.swift:3:22: note: to match this opening '('
let package = Package(
                     ^
/tmp/tmpw6hpbp41/swift-install/package/Package.swift:11:21: error: type '[Package.Dependency]' has no member 'package'
    dependencies: [.package(url: \"https://github.com/JustHTTP/Just\", from: \"0.7.1\"),
                    ^~~~~~~
Install Error: swift-build returned nonzero exit code 1.
19

Looks like you have wrong .package(...) string. I think there’s no need to escape double quotes.

20

@vova same issue with the following options

%install '.package(url: "https://github.com/JustHTTP/Just", from: \"0.7.1\")' Just

%install .package(url: "https://github.com/JustHTTP/Just", from: \"0.7.1\") Just