New feature: Installing SwiftPM packages in Jupyter


(Marc Rasi) #1

Hi!

I just implemented very basic support for installing SwiftPM packages in Jupyter: https://github.com/google/swift-jupyter/pull/45

You can try it out by checking out the branch, rerunning the register.py command, and then using the %install directive as described in the README.

It has some big limitations (also described in the README), and I’m interested in hearing from @jeremy which ones are the most important to fix.


(Jeremy Howard (Admin)) #2

Great start! The only show-stopper issue is the lack of C support. Many of the libs we’ll need for the course will have C pieces, I expect, and we’ll also want to show how to use C libs in your own packages.

BTW, this would be a good package to test the existing functionality would be swiftgd: https://github.com/twostraws/SwiftGD . Hopefully it’ll work fine already, since that’s only using system modules.


#3

Just tested it and it worked great (not the DeckOfPlayingCards example though, this one gave me an error). On our first notebook with the libs Path and Just though, using the first cell

%install '.package(url: "https://github.com/mxcl/Path.swift", from: "0.16.1")' Path
%install '.package(url: "https://github.com/JustHTTP/Just", from: "0.7.1")' Just

worked like a charm.

The error I get with the DeckOfPlayingCards example is:

Installing packages:
	.package(url: "https://github.com/NSHipster/DeckOfPlayingCards", from: "4.0.0")
		DeckOfPlayingCard
Fetching https://github.com/NSHipster/DeckOfPlayingCards
Fetching https://github.com/apple/example-package-fisheryates.git
Fetching https://github.com/nshipster/example-package-playingcard.git
Completed resolution in 1.29s
Cloning https://github.com/NSHipster/DeckOfPlayingCards
Resolving https://github.com/NSHipster/DeckOfPlayingCards at 4.0.0
Cloning https://github.com/apple/example-package-fisheryates.git
Resolving https://github.com/apple/example-package-fisheryates.git at 2.0.5
Cloning https://github.com/nshipster/example-package-playingcard.git
Resolving https://github.com/nshipster/example-package-playingcard.git at 4.0.0
'jupyterInstalledPackages' /tmp/tmpbzzssyaj: error: product dependency 'DeckOfPlayingCard' not found
warning: dependency 'DeckOfPlayingCards' is not used by any target
Install Error: swift-build returned nonzero exit code 1.

(Jeremy Howard (Admin)) #4

Having to reinstall packages every time you use them in Jupyter would be the other high priority issue to fix. It’s certainly usable without that - but especially since Swift notebooks need to be restarted so often at the moment, this is going to test one’s patience!


(Marc Rasi) #5

For the DeckOfPlayingCards example, I think you’re missing an “s” at the end of the “%install” directive.


#6

Eh! Thanks for catching, I need to learn to copy-paste properly :wink:


(Marc Rasi) #7

I have created a PR that makes it able to install packages with C: https://github.com/google/swift-jupyter/pull/48 (note: it requires a S4TF toolchain built on March 20 or later, specifically this commit).

I tried it out myself on BaseMath and SwiftGD, and I was able to import them and do simple operations. There is no inline display of SwiftGD images, but that would be pretty cool, and wouldn’t be too hard to implement.

Next, I’m going to see if I can get it to reuse its work when you restart the kernel.


(Jeremy Howard (Admin)) #8

@marcrasi I’m thinking for image transformations we should make it possible to use OpenCV. We used that for the older fastai 0.7 but problems in multiprocessing in python and opencv meant we had to move away from it. But in Swift we can use proper threads, so we won’t have this problem.

There is a ‘LegoCV’ project wrapping OpenCV in Swift, but it doesn’t have an SPM version and doesn’t mention Linux compatibility.

So perhaps creating and using a minimal OpenCV wrapper would be a good thing to do next with your new C-capable package installer? Note that using an optimized OpenCV build (i.e. march=native, or similar) is important - SIMD is critical for being able to process images fast enough to keep the GPU busy.

Also, could you check that your SwiftGD is able to use libjpeg-turbo successfully? Without that, the jpeg loading tends to take more time than everything else combined!