I’ve been trying to work with Harebrain on Google Colab and use Google Drive for storage. I tried using the python package, but the package is using a python package called pexpect, which displays a prompt / textfield input to accept the Google Drive auth token and this textfield is not displayed in the jupyter notebook. The input field simply isn’t included in the HTML output of the notebook.
This is the code I use (assuming you have the Stringextension for the shell method):
import Python
public let colab = Python.import("google.colab")
colab.drive.mount("/content/gdrive", force_remount: true)
let root_dir = "/content/gdrive/My Drive/"
let base_dir = root_dir + "fastai-v3/course-v3/nbs/dl2/"
("cd " + base_dir).shell()
I would like to contribute by fixing this, but I have trouble finding the right forum for this. On GitHub swift-jupyter has no issues and doesn’t point me anywhere else (e.g. Google groups). So my best guess what to bring this up here and hope to be pointed somewhere useful. Thanks in advance!
I have tried to mount gdrive from Python3 notebook. But, I can not see the folder on a Swift notebook. I also checked the google.colab.drive.mount. There is not parameter to pass the key. The signature is
Looks like Colab has recently switched to having a separate VM per notebook, so the strategy of mounting the drive on a Python notebook and then using it from the Swift notebook isn’t working any more, as you have noticed.
So you’re going to have to execute the Python drive mounting code from the Swift notebook. Since prompting the user for input doesn’t work in swift-jupyter, you’ll have to work around that somehow. (Making prompts work is probably very difficult or impossible, see the discussion of jupyter widgets in this).
Here’s an idea for how to make a workaround. Write a custom auth.authenticate_user() logic that does the work in two parts:
First have a function that does the work to get the authentication URL, and that returns that URL. (get_auth_url())
Write a second function that takes the code as an argument and sets up all the necessary information (save_auth_code(code))
Then, use PyDrive similarly to how it’s described in the Colab Python notebook, except instead of using auth.authenticate_user(), call get_auth_url(), visit the url, get the code, and finally call save_auth_code(code).
If someone writes something like this, it could be packaged up into a library so that others can use it. Maybe the Colab team would even be interested in having these improvements included in https://github.com/googlecolab/colabtools itself so that all clients have the option of authenticating without a textbox prompt.
It may not be worth the effort to try and get running on Swift Colab. It doesn’t support GPU even though you can select it so your own computer is going to be just as fast and most likely faster. You can see this by running the TensorFlow command below.
Sorry, I wasn’t paying attention to that issue. I investigated and I posted a workaround that I discovered. Also I think I see the right approach for properly fixing it, but it’ll take a week or two to get that released into Swift in Colab.
Google has a facility called a service account. This is designed for inter server communication. You create one using console.developers.google.com. You download a JSON.
Inside you will find an email, private key, and information to determine the public key.
Returning to your normal Google Drive you create a share quoting the service account email.
We then paste the JSON into a variable. This creates a security issue because you have just shown your private key.
The next step is to request a access token from Google which lasts for one hour maximum.
This is achieved by taking the email address and SHA256 and signing with the private key.
Google has the public key so verifies your request.
You can then request directory listings of the shared directory, download, upload and delete.
So your swift program can download from Google Drive and store on the local file. Process with Deep Learning and copy the output back to your Google Drive.
I have tested this with a Google Colab session and it worked.
There’s no longer a need for any workaround. Swift-Colab, the successor to google/swift-jupyter, fully supports Google Drive integration. The easiest way to mount is in the same cell that installs Swift-Colab.
!curl "https://raw.githubusercontent.com/philipturner/swift-colab/release/latest/install_swift.sh" --output "install_swift.sh"
!bash "install_swift.sh" "5.6.1" #// Replace 5.6.1 with newest Swift version
#// After this command finishes, go to Runtime > Restart runtime.
#// Your code that was appended to this cell:
from google.colab import drive
drive.mount("/content/drive")
This works because you’re still running the built-in Python kernel. Thus, we can bypass the fact that the Swift kernel doesn’t support Google Drive integration. Once you restart the runtime, it switches to the JIT-compiled Swift kernel. If you restarted it prematurely, you’re covered as well. Just switch back to Python mode and mount the drive:
// Run this cell in Swift mode
%system echo "python" > "/opt/swift/runtime"
// Then restart the runtime
#// Run this cell in Python mode
from google.colab import drive
drive.mount("/content/drive")
!echo "swift" > "/opt/swift/runtime"
#// The restart the runtime
Regarding the first post on this thread, I haven’t identified whether the root cause involves pexpect. I switched to using that library in Swift-Colab v2.1, and it allows much better I/O. But I did not create a mechanism for sending input to a process.
If anyone’s willing to investigate further, here’s the crash from attempting to install Google Drive while in Swift mode.
Now, there’s not even a need to use the Python Jupyter kernel to access Google Drive. I mounted a Google Drive to a notebook in Swift using PythonKit. Here’s how I did it.
The notebook usually crashes while calling into a nonexistent in-process kernel. The culprit is an invocation of _message.blocking_request. So, I set up a two-way messaging system between the LLDB and JupyterKernel threads. I swapped out the google.colab._message.blocking_request object using the syntax below. LLDB forwards the message’s raw JSON to the Jupyter kernel, which handles the request and sends it back to LLDB.
@_cdecl("redirect_stdin")
public func redirect_stdin() {
let _message = Python.import("google.colab")._message
_message.blocking_request = PythonFunction { args, kwargs in
// Encode `args` and `kwargs`
let input = encode_blocking_request(...)
KernelPipe.send(input, to: .jupyterKernel)
while true {
usleep(50_000)
// Read from the messaging pipe, see if messages exist.
let messages = KernelPipe.recv(from: .jupyterKernel)
if messages.count == 0 {
continue
}
return try decode_blocking_request(messages[0])
}
}.pythonObject
}
