Adding NBDev to an existing private repo: GITHUB_TOKEN

nbdev
1

The docs say to set the environment variable GITHUB_TOKEN with a personal access token (PAT) before using nbdev_new to add NBDev to an existing private github repository. How do you do that?

Generating the PAT is fine, from github docs, but what do you do with it afterwards? From other docs it reads like you need to create a github Actions workflow to use the token… but wouldn’t that mean you’re putting a piece of sensitive data online? Or are workflows more secure? Also if you have to create a workflow to set the token, then wouldn’t that not help for NBDev since it creates its own workflows?

Taking a look at a workflow @hamelsmu linked in another thread, the token is assigned as a variable – not copy-pasted. So is the token read from your account automatically? Or do you have to set it from the command-line? Or is that the default token that won’t work for private repos?

2 Likes
2
1 Like
3

The section that says “Creating encrypted secrets for an environment” – that isn’t the same environment as the environment variable mentioned in the NBDev setup docs is it? And an ‘encrypted secret’ is not the same thing as a personal access token?

I didn’t have the option to create an environment secret in any case, so I made a respository one. But every example of actually using the ‘secret’ in the docs looks something like:

steps:
  - name: Hello world action
    with: # Set the secret as an input
      super_secret: ${{ secrets.SuperSecret }}
    env: # Or as an environment variable
      super_secret: ${{ secrets.SuperSecret }}

and if I understand correctly, that text would be in a workflow/‘actions’ file … but there isn’t a workflow or action for NBDev yet since I haven’t run it … and I’m not supposed to run it until that ‘secret’ is in use. I’m having a chicken-and-the-egg problem here.

Now when I try running nbdev_new in my repository, it gives an error and says I was supposed to give a name. Going up a directory and using the name gives another error since the directory already exists. Also the docs say to just run it inside so that couldn’t be right. This is very confusing.

Also, this is for a different variable than the GITHUB_TOKEN mentioned in the NBDev docs? So, to get this working, we need to setup two variables?

4

The prose is a requirement to maintain sanity. The solution is at the bottom.

Last episode nbdev_new refused to run, not because it couldn’t find the private repo like the first time, but because it wanted a name:

REPONAME % nbdev_new               
usage: nbdev_new [-h] [--template_git_repo TEMPLATE_GIT_REPO] name
nbdev_new: error: the following arguments are required: name

Simple Solutions

So I searched and started thinking: maybe I should just put one block on top of another instead of trying to push a button? IE: could setting up NBDev be as simple as drag and dropping the template? This post by J. Howard suggests so. Unfortunately it’s over a year old so here’s hoping nothing changed.

The Confusion

A couple thoughts here on what made this all so confusing. The docs say create a PAT. The link from Husain, who made/works on the entire github actions/workflow thing at the root of this, suggests to use an “encrypted secret”. Do we use both? I assume only the second. But the link shows 3 different types of encrypted secret. No word on which is appropriate. Also the PAT linked from the docs: GitHub’s docs say it only works with HTTPS repos… so if you tried to have good security and use SSH it won’t work? That doesn’t sound good. The biggest confusion is what looks like a cold-start / chicken-and-egg situation. There’s no github action/workflow until NBDev makes one, but NBDev can’t make one if it can’t find the repository… but it won’t be able to find the repo if there’s no secret key in the action/workflow… hence the confusion.

As a recap, what I did first was create a PAT, and when that didn’t work I made a repo encrypted secret. I don’t know how you connect those variables to NBDev, and the error message NBDev gives in the terminal (in the repo’s root directory, without any arguments) is that it’s missing a name.

unless what you’re supposed to do is copy-paste nbdev_template into your repo, then go into its workflow file (.github/workflows/main.yml) and add the PAT there. Though what I’m understanding now is the PAT is made obsolete by a repository encrypted secret, so maybe you put that in via its $VARIABLENAME instead?

Trying to go backwards

I also wondered… the new failure mode (nbdev_new wanting a name) started after I added a personal access token (PAT) as per the NBDev docs and a repository encrypted secret as per the suggestions in Husain’s link above. Hard deleting the local repo directory via rm -rf and re-cloning didn’t get the first failure mode (nbdev_new unable to find the private repo) back, and I tried twice to make sure. I didn’t edit any workflow / actions files yet so I don’t know where an interaction would be coming from, but if that’s the only thing I changed: what if I deleted the PAT and the repo encrypted secret?

→ No, deleting either or both changed nothing. Likely because I didn’t put them in a workflow file yet.

Solution

Goal: integrate NBDev into a private repository

Issue: the nbdev_new command cannot find private repositories.

  1. copy-paste the nbdev_template repository into your private repo.
    • Exceptions: omit the .git folder; add the contents of .gitignore to your own.
  2. add a “_data” folder in the “docs” folder.
    • Thanks to enryH & paulkarayan in this issue thread. Otherwise when running nbdev_build_lib NBDev won’t be able to create the “topnav.yml” file. (I didn’t have this problem last time I tried using NBDev a year or so ago, and I made sure to be up to date with conda install -c fastai nbdev. I also think you have to delete topnav.yml if you change your repo’s name.)
  3. Continue with normal NBDev setup: running nbdev_install_git_hooks and editting settings.json.

This likely doesn’t solve NBDev not being able to find the repo, and I expect that’ll break workflows / actions, so you’ll have to look into how to add either a PAT or Encrypted Secret environment variable to NBDev’s workflow file. But this does get NBDev basically working. If/when I find out how to get the variables working, I’ll update this post. And if anyone knows how it works, please feel free to share.

5

Can you DM me so I can pair with you live to see the problem? I would love to follow along in real time so I can then improve the docs. Thanks

2 Likes
6

It looks like my advice may have been incorrect and my apologies for that- if you could set a local environment variable GITHUB_TOKEN believe that would have done the trick

Sorry for the trouble

1 Like
7

Oh not at all. The issue is I need to learn how github actions and workflows work :sweat_smile:. I’m going to chip away at this more until it’s working smoothly. I’ll PR an update to the docs once I find a clean solution.