Help test new exporter

(Jeremy Howard (Admin)) #1

Many thanks to @pcuenq there is a new nb module exporter to try out:

Please folks give it a go and let us know if it works for you! :slight_smile:

1 Like

(Jeremy Howard (Admin)) #2

@pcuenq how do we use this to export notebooks that depend on other notebooks? And how do we import such notebooks? Is that all automatic? Do we have to manually import every notebook namespace that’s used?

0 Likes

(Pedro Cuenca) #3

It’s similar to the current system, but package dependencies are automatically identified and there is no need to run a python script afterwards. You just need to import the last notebook you saved, declarations in previous notebooks are inherited.

Let me try a brief walkthrough.

  • In the first notebook you want to export (say, test_00.ipynb), add the NotebookExport package as a %install dependency, alongside any others you use:
%install-location $cwd/swift-install
%install '.package(url: "https://github.com/mxcl/Path.swift", from: "0.16.1")' Path
%install '.package(url: "https://github.com/latenitesoft/NotebookExport", .branch("fastai"))' NotebookExport
  • Define your exportable cells as usual:
//export
public func test_00() -> String {
    return "Hello from the test_00 notebook"
}
  • Export your notebook like this:
import Path
import NotebookExport

let exporter = NotebookExport(Path.cwd/"test_00.ipynb")
print(exporter.export())

If everything went well, that should print a success message. A Swift package called ExportedNotebook_test_00 will have been created, and that package can be imported in subsequent notebooks. I chose not to use the FastaiNotebook_ prefix we were using to prevent conflicts or confusion. You can change it if you want to.

  • In the second notebook (test_01.ipynb), install the first one. Note you don’t need to install the exporter this time, since it was saved as a dependency:
%install-location $cwd/swift-install
%install '.package(path: "$cwd/ExportedNotebook_test_00")' ExportedNotebook_test_00
  • Import the notebook and use the code:
import ExportedNotebook_test_00
test_00()
  • Add new //exportable code, and save the notebook in the same fashion:
//export
public func test_01() -> String {
    return "This is notebook test_01"
}
import Path
import NotebookExport

let exporter = NotebookExport(Path.cwd/"test_01.ipynb")
print(exporter.export())
  • Repeat the same steps in the third notebook. Note that you can refer to notebook 0’s declarations simply by importing notebook 1:
%install-location $cwd/swift-install
%install '.package(path: "$cwd/ExportedNotebook_test_01")' ExportedNotebook_test_01
// Uses the sources from ExportedNotebook_test_00 too
import ExportedNotebook_test_01
print(test_00())
print(test_01())

Please, let me know about any bugs, comments or suggestions!

3 Likes

#4

I have fully tested it on all the notebooks and confirmed it’s working. Just switched the entire dev_swift folder to it.

1 Like

(Pedro Cuenca) #5

There’s a problem, though. Dependency paths are resolved, so the paths in your machine do not match mine’s. We need to keep them relative, otherwise people will have to re-export the lessons to follow them. Working on a fix.

Sorry about that :cry:

0 Likes

#6

In my case removing the swift-install folder seemed to do the trick?

0 Likes

(Pedro Cuenca) #7

No, the dependency path looks like /home/ubuntu/dev_swift/FastaiNotebook_03_minibatch_training, but it should be ../FastaiNotebook_03_minibatch_training.

Do you want me to push a PR with the paths changed manually while I finish the fix?

0 Likes

#8

Yes please!

0 Likes

(Pedro Cuenca) #9

Submitted.

2 Likes

(Pedro Cuenca) #10

I submitted version 0.3.0 to the repo with this fix. I’ll retest all the notebooks again and will check that the exported packages can be imported in a different directory.

Sorry once again for the oversight.

0 Likes

#11

Eh that’s no problem, you’re fixing it super fast :wink:

1 Like

#12

Mmmm, I can’t export a notebook that’s already exported without deleting the folder of the package.

0 Likes

#13

Ah, also we still need to copy the updated swift files in all the other folder (or run the export on all the notebooks after to make that update).

0 Likes

(Jeremy Howard (Admin)) #14

Maybe we want a “create_all” function and just always run that instead.

1 Like

(Pedro Cuenca) #15

This is working for me. Is it giving you an error or just not regenerating the file?

0 Likes

(Pedro Cuenca) #16

Yes, I’m using hard links but that’s a half-baked way to support that. They get converted to regular files when you upload them to git, of course. I’ll do something similar to what @jeremy suggested.

And I’ll change the default prefix to FastaiNotebook_ so we don’t need to type it.

1 Like

#17

It’s giving me an error message.

0 Likes

(Jeremy Howard (Admin)) #18

@GuggerSylvain what did you do and what error did you get?

0 Likes

#19

I ran

import NotebookExport
let exporter = NotebookExport(Path.cwd/"09_optimizer.ipynb")
print(exporter.export(usingPrefix: "FastaiNotebook_"))

and got

failure(reason: "The operation could not be completed")

This disappears if I remove the directory of the package.

0 Likes

(Pedro Cuenca) #20

I believe this is happening while copying the sources from previous notebooks. I still need to rework that part as we discussed, but meanwhile, can you please use version 0.4.0 I just pushed?

%install-location $cwd/swift-install
%install '.package(url: "https://github.com/latenitesoft/NotebookExport", from: "0.4.0")' NotebookExport

Let me know if that works. I’ll improve error messages too.

0 Likes