Workflow run jupyter notebooks and update fastpages

nbdev
1

@hamelsmu (I’ve just started using fast pages and absolutely love it - thanks for all your work on it.)

Is there any documentation on setting a scheduled github action to refresh a jupyter notebook and refresh fast pages blog? I know that you’ve set this up for the covid tracker github page. I’ve attempted to copy your update-nb.yaml file and adapt it for my project/blog (and the run_notebooks.sh file which I haven’t changed at all).

My yaml file is here

I modified the yaml file so that it only run runs on schedule. I.e:

name: Update Notebooks And Refresh Page
on:
  schedule:
      - cron:  '0 3 * * *'

I’ve removed some of the code if it was written to run on non-schedule events. I’ve left the GITHUB_TOKEN and SSH_DEPLOY_KEY code in there but I think that’s fine. I’ve also got a ‘github-actions[bot]’ in the code and not sure if that causes problems for running on my account.

I’ve read the github actions documentation, but am not that familiar with .yaml files in general and have only just started using github. So apologies if this is a basic question, and an open-ended one. I’m still learning how to write good questions.

What I could benefit from are the following:
1 - any links to docs on automating running of jupyter nbs on github (either as part of fast pages or stand alone)
2 - any links to docs on how to debug git hub workflows - I just get an ‘exit code 1’ and don’t know what to do with it.
3 - and of course any tips on how I can get to a solution!

Alex

2

I don’t have any docs. Feel free to closely examine the github workflows in https://github.com/github/covid19-dashboard/blob/master/.github/workflows/update-nb.yaml

This is the shell script I invoke from the Actions workflow. https://github.com/github/covid19-dashboard/blob/master/_action_files/run_notebooks.sh

This action may help you debug https://github.com/mxschmitt/action-tmate

If you end up making something, you should write a blog post about it!

3

Thank you very much. Yes - those are the yaml and sh files that I’ve copied. I’ll check out the action-tmate action as well. Good suggestion re the blog post!

4

Thanks, Hamel and Alex! I used the workflow template and the shell script and was able to set up an auto updating notebook very quickly. I wrote up my notes here: https://sanzgiri.github.io/jupyter/2020/04/15/fastpages-jupyter-notebooks.html

Question - what are the limits on the compute available to run the notebooks? and what cloud do they run in?

1 Like
5

Please search the docs for GitHub Actions (I don’t know the answer off hand)

6

Found it here: https://help.github.com/en/actions/getting-started-with-github-actions/about-github-actions#usage-limits

With the 6 hr job execution limit and the ease of scheduling - I can think of lots of use cases :smile:

7

I think I need to look at the required packages I’ve listed - that’s probably where the error is coming from. The other thing I need to do is run the auto updating for specific notebooks. I think there is code from @hamelsmu workflow that I can copy for this.

Thanks for doing @akrimedes - shows that I’m close in any case.

8

@alexxcollins21 any places that you are stuck currently? Can you share a little bit more about your project?

9

I haven’t actually tried to get automatic updates working since I asked you the question. I’ve been too busy writing a neural net from scratch to label the MNIST data set!

One thing I have done though is finally sort out the mess of python installation, virtual environments and packages on my notebook. Since starting to code I hadn’t settled on a standard package installation method and my computer was a total mess. I uninstalled everything and started again with Miniconda and Conda environments. One benefit of this is a very simple process to create an environment.yml file which I hope will help with this refresh notebook process and would have made my implementation of voila/binder app much quicker too.

Thanks for your help and I’ll be sure to update you once I’ve got it working.The dashboard I created is here. I used altair for the charts - it’s the first time I’ve used altair but I really like the visualisations.

10

Hello, about this topic I have a question. I’m trying to update my notebooks using the workflow template and the shell script. But I got an error to run Jekyll build and when I execute the papermill in my machine, the plotly doesn’t show anything. Anyone got this error?