Python textfield output not working

(David Pfahler) #1

I’ve been trying to work with Harebrain on Google Colab and use Google Drive for storage. I tried using the python package, but the package is using a python package called pexpect, which displays a prompt / textfield input to accept the Google Drive auth token and this textfield is not displayed in the jupyter notebook. The input field simply isn’t included in the HTML output of the notebook.

This is the code I use (assuming you have the String extension for the shell method):

import Python
public let colab = Python.import("google.colab")
colab.drive.mount("/content/gdrive", force_remount: true)
let root_dir = "/content/gdrive/My Drive/"
let base_dir = root_dir + "fastai-v3/course-v3/nbs/dl2/"
("cd " + base_dir).shell()

I would like to contribute by fixing this, but I have trouble finding the right forum for this. On GitHub swift-jupyter has no issues and doesn’t point me anywhere else (e.g. Google groups). So my best guess what to bring this up here and hope to be pointed somewhere useful. Thanks in advance!

0 Likes

(Jon Tow) #2

Hey @davidpfahler. A good place to ask these questions is the swift for tensorflow google group. They’re very helpful over there so give it a try!

0 Likes

(David Pfahler) #3

Thanks a lot! I will certainly do that.

Update: For anyone interested, see this thread on the swift for tensorflow google group.

0 Likes

(Qin Lu) #4

Hello @davidpfahler,

I have tried to mount gdrive from Python3 notebook. But, I can not see the folder on a Swift notebook. I also checked the google.colab.drive.mount. There is not parameter to pass the key. The signature is

def mount(mountpoint, force_remount=False, timeout_ms=60000)

Have you found a way to solve this issue?

Thank you.

0 Likes

How to run a script shell in google colab for Swift?
(Marc Rasi (S4TF Team)) #5

Looks like Colab has recently switched to having a separate VM per notebook, so the strategy of mounting the drive on a Python notebook and then using it from the Swift notebook isn’t working any more, as you have noticed.

So you’re going to have to execute the Python drive mounting code from the Swift notebook. Since prompting the user for input doesn’t work in swift-jupyter, you’ll have to work around that somehow. (Making prompts work is probably very difficult or impossible, see the discussion of jupyter widgets in this).

Here’s an idea for how to make a workaround. Write a custom auth.authenticate_user() logic that does the work in two parts:

  1. First have a function that does the work to get the authentication URL, and that returns that URL. (get_auth_url())
  2. Write a second function that takes the code as an argument and sets up all the necessary information (save_auth_code(code))

Then, use PyDrive similarly to how it’s described in the Colab Python notebook, except instead of using auth.authenticate_user(), call get_auth_url(), visit the url, get the code, and finally call save_auth_code(code).

If someone writes something like this, it could be packaged up into a library so that others can use it. Maybe the Colab team would even be interested in having these improvements included in https://github.com/googlecolab/colabtools itself so that all clients have the option of authenticating without a textbox prompt.

3 Likes

(David Pfahler) #6

I have focused on other things since and haven’t pursued this. Sorry I can’t help, but didn’t want to leave you without a response.

1 Like

(Stephen Johnson) #7

It may not be worth the effort to try and get running on Swift Colab. It doesn’t support GPU even though you can select it so your own computer is going to be just as fast and most likely faster. You can see this by running the TensorFlow command below.

_ExecutionContext.global.deviceNames
▿ 2 elements
  - 0 : "/job:localhost/replica:0/task:0/device:CPU:0"
  - 1 : "/job:localhost/replica:0/task:0/device:XLA_CPU:0"
1 Like

(Jeremy Howard (Admin)) #8

@marcrasi I wasn’t aware of this - is this a known problem?

0 Likes

(Stephen Johnson) #9

I posted this issue in the swift-apis github a month ago but no response.

0 Likes

(Marc Rasi (S4TF Team)) #10

Sorry, I wasn’t paying attention to that issue. I investigated and I posted a workaround that I discovered. Also I think I see the right approach for properly fixing it, but it’ll take a week or two to get that released into Swift in Colab.

1 Like

(Stephen Johnson) #11

Thank you. Saw your workaround and am going to give it a try today. Very appreciated!!

0 Likes

#12

Hi David

Swift and Google Drive

I have an alternative solution.

Google has a facility called a service account. This is designed for inter server communication. You create one using console.developers.google.com. You download a JSON.
Inside you will find an email, private key, and information to determine the public key.
Returning to your normal Google Drive you create a share quoting the service account email.

We can now return to google colab. We add to the first line
%install ‘.package(url: “https://github.com/IBM-Swift/Swift-JWT.git”, from: “3.0.0”)’ SwiftJWT

we also import FoundationNetworking and SwiftIBM.

We then paste the JSON into a variable. This creates a security issue because you have just shown your private key.

The next step is to request a access token from Google which lasts for one hour maximum.

This is achieved by taking the email address and SHA256 and signing with the private key.

Google has the public key so verifies your request.

You can then request directory listings of the shared directory, download, upload and delete.

So your swift program can download from Google Drive and store on the local file. Process with Deep Learning and copy the output back to your Google Drive.

I have tested this with a Google Colab session and it worked.

Regards Conwyn

1 Like