Can't import SwiftVips

(Jeremy Howard (Admin)) #1

I’ve just starting adding libvips for the C interop example, which you can see in nb sox_integration_example.ipynb. First you’ll need to install libvips, following SwiftVips/install.sh docs.

After I import SwiftVips or import vips I get the dreaded expression failed to parse, unknown error. @marcrasi any chance you could take a look since this is a blocker for this part of the next lesson? @pcuenq do you have any insights on what might be happening here?

0 Likes

(Jeremy Howard (Admin)) pinned #2
0 Likes

(Pedro Cuenca) #3

Just saw this post, installing libvips right now. Tagging @vova in case he has any ideas.

0 Likes

(Pedro Cuenca) #4

I got a previous error while linking. String(cString: dlerror()) reports that libvips.so.42 cannot be found, but it does exist in my machine. Did you have that error?

0 Likes

(Jeremy Howard (Admin)) #5

I don’t have any errors using it from a CLI app.

0 Likes

(Pedro Cuenca) #6

Me neither. But in the notebook I get this error when running the install cell:

Install Error: dlopen error. Run `String(cString: dlerror())` to see the error message.
0 Likes

(Jeremy Howard (Admin)) #7

I didn’t see that. I’ll try updating my swift-jupyter.

0 Likes

(Pedro Cuenca) #8

I got rid of that but I still get the parse error. What I’ve seen so far is that libvips depends on glib and glib-object. I’ve tried to define explicit system library dependencies for them; or link them inside the vips module map file, but I’ve had no success so far.

0 Likes

(Jeremy Howard (Admin)) #9

Interesting. I had a similar problem when I tried to use a glib-requiring lib a while ago and didn’t use spm - instead I tried to use an objc bridging header. I never figured out how to get it working.

0 Likes

(Pedro Cuenca) #10

More specifically, this is the error message from the REPL:

Welcome to Swift version 5.0-dev (LLVM dcb9eb74a7, Clang 95cdf7c9af, Swift ec27d9b99d).                                
Type :help for assistance.                                                                                             
  1>                                                                                                                   
  2>                                                                                                                   
  3>                                                                                                                   
  4> import Glibc                                                                                                      
  5> let h = dlopen("package/.build/x86_64-unknown-linux/debug/libjupyterInstalledPackages.so", RTLD_NOW)              
h: UnsafeMutableRawPointer? = (_rawValue = 0x000000000134cb70)                                                         
  6> import vips                                                                                                       
warning: Swift error in fallback scratch context: <module-includes>:1:10: note: in file included from <module-includes>
:1:                                                                                                                    
#include "/home/pedro/code/s4tf/fastai_docs/dev_swift/SwiftVips/Sources/vips/shim.h"                                   
         ^                                                                                                             
                                                                                                                       
/home/pedro/code/s4tf/fastai_docs/dev_swift/SwiftVips/Sources/vips/shim.h:1:10: note: in file included from /home/pedro
/code/s4tf/fastai_docs/dev_swift/SwiftVips/Sources/vips/shim.h:1:                                                      
#include <vips/vips.h>                                                                                                 
         ^                                                                                                             
                                                                                                                       
error: /usr/local/include/vips/vips.h:87:10: error: 'glib.h' file not found                                            
#include <glib.h>                                                                                                      
         ^                                                                                                             
                                                                                                                       
error: repl.swift:6:8: error: could not build C module 'vips'                                                          
import vips                                                                                                            
       ^                                                                                                               
                                                                                                                       
                                                                                                                       
                                                                                                                       
note: This error message is displayed only once. If the error displayed above is due to conflicting search paths to Cla
ng modules in different images of the debugged executable, this can slow down debugging of Swift code significantly, si
nce a fresh Swift context has to be created every time a conflict is encountered.                                      
                                                                                                                       
expression failed to parse, unknown error                                                                              

I’m guessing there are some system module maps that are not being correctly prepared in the install directory. I tried to force it by defining pkg-config dependencies for them, but it didn’t work. I’ll see if I can get some info from the build.db database, maybe we can whitelist them.

1 Like

(Jeremy Howard (Admin)) #11

On my system that’s in /usr/include/glib-2.0. Maybe we just do a symlink to /usr/include?

0 Likes

(Jeremy Howard (Admin)) #12

Here’s how to get the info with pkg-config: pkg-config --cflags --libs glib-2.0

0 Likes

(Jeremy Howard (Admin)) #13

Well this is interesting:

$ pkg-config --cflags --libs vips
... -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include ... -lgobject-2.0 -lglib-2.0

So why aren’t those headers available? Shouldn’t they have been compiled in to the swiftmodule somehow? I don’t understand how swiftmodules work - but I thought they had the symbol info in them?..

0 Likes

(Pedro Cuenca) #14

That’s exactly what I thought, I don’t understand why the headers are needed at what should just be runtime.

0 Likes

(Jeremy Howard (Admin)) #15

More info - looking at SwiftVips/.build/debug.yaml I see:

other-args: ["-target","x86_64-unknown-linux","-swift-version","4.2","-enable-batch-mode","-index-store-path","/home/jhoward/git/fastai_docs/dev_swift/SwiftVips/   .build/x86_64-unknown-linux/debug/index/store","-sdk","/","-Onone","-g","-enable-testing","-j64","-DSWIFT_PACKAGE","-DDEBUG","-Xcc","-fmodule-map-file=/home/jhoward/   git/fastai_docs/dev_swift/SwiftVips/.build/x86_64-unknown-linux/debug/CSwiftVips.build/module.modulemap","-I","/home/jhoward/git/fastai_docs/dev_swift/SwiftVips/Sour   ces/CSwiftVips/include","-Xcc","-fmodule-map-file=/home/jhoward/git/fastai_docs/dev_swift/SwiftVips/Sources/vips/module.modulemap","-I/usr/local/include","-I/usr/inc   lude/glib-2.0","-I/usr/lib/x86_64-linux-gnu/glib-2.0/include","-module-cache-path","/home/jhoward/git/fastai_docs/dev_swift/SwiftVips/.build/x86_64-unknown-linux/deb   ug/ModuleCache","-Xfrontend","-color-diagnostics"]

So the -I flags appear there. But no other of the -I flags from pkg-config vips are there. So it seems like these need to be part of the compilation.

0 Likes

(Pedro Cuenca) #16

I cannot believe this, but the following REPL invocation works:

swift -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include

Then dlopen and import vips succeed.

Looking at ways to hack it into the kernel. (I tried to use %install-swiftpm-flags unsuccessfully).

1 Like

(Pedro Cuenca) #17

I can’t get it to work inside the kernel.

@marcrasi I have replaced None in the following code with a list of arguments extracted from the yaml file:

   self.process = self.target.LaunchSimple(None,    
                                           repl_env,     
                                           os.getcwd()) 

According to the source, the arguments has to be a list. Is there another way you know of to pass command-line arguments to the swift process? Or is repl_swift perhaps ignoring them?

0 Likes

(Pedro Cuenca) #18

@jeremy This ugly workaround does indeed work in my system:

cd /usr/include
for f in /usr/include/glib-2.0/*; do sudo ln -s $f .; done
for f in /usr/lib/x86_64-linux-gnu/glib-2.0/include/*; do sudo ln -s $f .; done

Nevertheless we need to figure out how to make it work inside the jupyter kernel, now that we know that there are additional arguments that need to be observed.

1 Like

(Marc Rasi (S4TF Team)) #19

It is not possible to pass commandline arguments to the swift compiler in the Jupyter kernel, because it invokes the compiler stages directly without going through any commandline flag processing code. (This “LaunchSimple” method passes commandline arguments to the “Swift program” itself, not to the compiler.)

It is just a few extra -I arguments that we need? The SWIFT_IMPORT_SEARCH_PATH env variable that I recently added (https://github.com/apple/swift-lldb/commit/bdb96e8b352f7cb18e2b3b66f2d3f75d92f81dcd) does the same thing as -I, except that it works in the Jupyter kernel. Unfortunately, it can only do one and it’s already being used for package installation. But I could add support for more pretty easily if that’s all we need.

^ Those are all the thoughts that I have without having actually tried this out myself. I will try it out myself Monday morning and see if I can figure anything out!

0 Likes

(Jeremy Howard (Admin)) #20

Yes. These extra args should be figured out from SPM, or using the same underlying method SPM uses.

0 Likes