Nbdev for private pypi repositories and poetry package management

At our company, we’re using a private PyPI server and managing dependencies using poetry. Has anyone managed to get either one of those working with nbdev?

I’m particularly concerned about the CI/CD pipelines as the installation steps are specified in fastai/workflows, it looks like specifying a different installation procedure would either require creating a modified local copy of the original action or to make a contribution to nbdev source.

1 Like

Is there a better way to structure the workflows for your use-case? How does yours need to be different? Specific flags used by pip, or different commands?

2 Likes

There are two issues that I conflated in my original post: private PyPI and poetry package manager instead of pip.

I’ll experiment more with this today. Do you think poetry support would be a welcome contribution?

Private PyPi

Installing dependencies from a private PyPi repository requires only an additional flag to pip and some secrets being passed.

So the following:

test -f setup.py && pip install -e ".[dev]"

Becomes:

test -f setup.py && pip install --extra-index-url https://$PYPIUSER:$PYPIPASSWORD@$PYPISERVER -e ".[dev]"

Poetry

The above would not suffice for our use case as we’re using poetry.
Installing dependencies with poetry would require using entirely different commands.

On the initial scan of the nbdev-ci workflow, it looks that the following commands would need to be replaced by their poetry equivalents. But there might be more.

        python -m pip install --upgrade pip
        if [ $USE_PRE ]; then
          pip install -U git+https://github.com/fastai/fastcore.git
          pip install -U git+https://github.com/fastai/ghapi.git
          pip install -U git+https://github.com/fastai/execnb.git
          pip install -U git+https://github.com/fastai/nbdev.git
        else
          pip install -U nbdev
        fi
        echo "Doing editable install..."
        test -f setup.py && pip install -e ".[dev]"
2 Likes

Option 1. Doing it without nbdev

How to publish a package to a private pypi server:

test -f setup.py && rm -rf dist build *.egg-info .eggs
python setup.py sdist bdist_wheel
twine upload --repository-url <URL> --username <PYPI_USERNAME> --password <PYPI_PASSWORD> dist/*

How to then install the package from a private pypi server:

pip install --extra-index-url "http://<PYPI_USERNAME>:<PYPI_PASSWORD>@123.134.56.7:8080" private_package

Change the placeholders:

  • <URL>: The URL of your private PyPI repository.
  • <PYPI_USERNAME>: The username for accessing your private PyPI repository.
  • <PYPI_PASSWORD>: The password for accessing your private PyPI repository.
  • 123.134.56.7: The placeholder IP address of your private PyPI repository.
    • Note: it’s better to use a domain like my-pip-server.com instead of an IP address so that you can set up an SSL connection, otherwise your credentials are passed in plain text over the internet. Instructions how to do that are available here and here.
  • 8080: The placeholder port number for your private PyPI repository.
  • private_package: The private package you want to install.

Option 2. Using nbdev to do it

nbdev itself doesn’t do anything different, as seen in their def release_pypi function in nbdev/18_release.ipynb at 3f0266328c2537a35487767288bc4283308f7048 · fastai/nbdev · GitHub

In fact, you probably can just create a ~/.pypirc file like

[distutils]
index-servers =
    private

[private]
repository: https://pypi.example.com
username: user
password: PUT_PASSWORD_HERE

and just pass release_pypi(repository="private") and you will be able to upload to your private PyPI server.

Existing GitHub issues related to this topic

1 Like