Automate hub installation

fastai
1

Many users use hub for doing PRs. Installing the binary hub takes several steps and it would be nice to automate this process and make it platform-independent, so that all the nuances are hidden behind the script. The installation process is described here. It’d be great to replace that process with just: run hub-install to install hub regardless of the platform.

This little project requires knowing both windows and linux (mac too, but not really, since it is the same as linux in this situation) so that the automated program could automatically derive the platform the user is on, fetch the right latest binary, unpack and install it into the right place - e.g. if it’s run from a conda-activated environment it’ll install it into that env’s bin directory, or perhaps user ~/bin/ or into a system-wide location via sudo.

It should be written in python to support windows users who may not have bash.

1 Like
2

Another option is to create a conda recipe. There’s a similar one already for cross-platform 7zip installation here:

You could use the existing hub binaries instead of having the recipe compile them, if that’s easier.

3

But if you re-package it you also have to maintain and update it. With an automatic installer there is no extra ongoing work to do.

4

Hi @stas

I made an attempt to solve the hub installation. Have been successful partially (Conda env based installation pending).
please find the gist located here

Tested out the installation on colab. See link

Kindly provide your feedback and inputs to handle conda environment.

P.S: I am not able to find the current conda env path in windows as there is no sed or grep module in windows. Also i am not sure about installation with admin privileges in the current script (Windows). sudo based installation throws error in colab.
~Gokkul

1 Like
5

Looking good, @gokkulnath - great work so far, thank you.

There should be no sudo if conda env is detected, since it’s under user dirs, and not system-wide.

I am not able to find the current conda env path in windows as there is no sed or grep module in windows.

Of course, you don’t need sed or grep, since now the following:

prefix=`which conda | sed 's/\/bin\/conda//'` ./install

should be done in python, using pathlib. You just need to get the location where conda is installed and install hub into the same location.

If conda is not to be found for now, leaving the default installer configuration to let it install system-wide. There are a few other virtual python systems, support for which can be added later if needed.

Unfortunately, I can’t help with windows, since I don’t know it. Perhaps, someone else with knowledge of windows can help. But if not, we will leave it linux/mac until someone with that kind of knowledge will help.

6

@gokkulnath, are you still planning to complete this?

7

Hi @stas,

Sorry! My Apologies … I got carried away with my personal side project so lost track of this work.
Regarding, the pending part …

The Following code snippet will do the same work.

if ‘CONDA_DEFAULT_ENV’ in os.environ.keys():
if debug : print(‘Script Executed inside Conda Environment’)
conda_path=Path(os.environ[‘CONDA_PREFIX’])
cmd = f’prefix={conda_path} {loc}/install’

I will post an updated Gist (sometime this week) once i verify it working on my local system and colab.

1 Like
8

Hi @stas,

Request you to review the code.
I had tested on the following:
Colab : works
Linux conda Env : works
On Windows based machine the script requires admin privileges to install. Haven’t figured out to install locally within conda environment. In Current approach, i believe installation is done system wide .
Local windows7 : works

Need someone to test on Windows10 based Machine.

Looking forward to hear comments/feedback/suggestions.
Please find the updated Gist. Here
Colab NB : Link

P.S: Looking forward to contribute more. Can you point me to any low hanging fruit tasks? Thanks.

~Gokkul

9

Awesome!

linux here:

git clone https://gist.github.com/21c05b2bb20e1a20c536c4160d1eaf7b.git
cd 21c05b2bb20e1a20c536c4160d1eaf7b/
python hub-install.py
Traceback (most recent call last):
  File "hub-install.py", line 80, in <module>
    conda_path=Path(os.environ['CONDA_PREFIX'])
NameError: name 'Path' is not defined

So I added:

from pathlib import Path

and try again:

Traceback (most recent call last):
  File "hub-install.py", line 91, in <module>
    subprocess.run(cmd.split(),shell=False, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/home/stas/anaconda3/envs/fastai/lib/python3.7/subprocess.py", line 466, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/stas/anaconda3/envs/fastai/lib/python3.7/subprocess.py", line 769, in __init__
    restore_signals, start_new_session)
  File "/home/stas/anaconda3/envs/fastai/lib/python3.7/subprocess.py", line 1516, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '`prefix=/home/stas/anaconda3/envs/fastai`': '`prefix=/home/stas/anaconda3/envs/fastai`'

So I fixed up the final bit of the script to be:

env = {}
#Call OS Specific Installation Script:
if os_type =='linux' or os_type =='mac':
    cmd=f'{loc}/install'
    # Check for Conda Based Environment
    if 'CONDA_DEFAULT_ENV' in os.environ.keys():
        if debug : print('Script Executed inside Conda Environment')
        env["prefix"] = os.environ['CONDA_PREFIX']
elif os_type=='windows':
    cmd=f'install.bat'

# Installation of hub library

print(f"running: {cmd} (env={env})")
my_env = {**os.environ, **env}
subprocess.run(cmd.split(),shell=False, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=my_env)

and then it works.

The env can’t be passed shell style, subprocess requires the executable first.

This is tested on linux

The updated code is here: https://gist.github.com/stas00/9045b2b7ad08ff861fd45f2576ccbec0

P.S: Looking forward to contribute more. Can you point me to any low hanging fruit tasks? Thanks.

Perhaps this? Port fastai-nbstripout to jupyter lab

10

Oops I missed my first line of import Statement. Thanks for fixing that.

Cool. I was not aware of that.

This One works fine for me… Tested locally and on colab notebook runs without any error.

i will have a look and update on that post. Thanks a lot.

11

I removed it subsequently, since I changed the code not to use it.

The env can’t be passed shell style, subprocess requires the executable first.

Cool. I was not aware of that.

Well, you can if you pass env as the command, but I don’t think it’d work on windows. I don’t know if it has it.

The updated code is here: https://gist.github.com/stas00/9045b2b7ad08ff861fd45f2576ccbec0

This One works fine for me… Tested locally and on colab notebook runs without any error.

So which one should we use to let people test - let’s have one place.

In general I think there should be a separate repo for git-tools, as this is not really fastai-specific. I started one here: https://github.com/stas00/git-tools with pr-branch tool, perhaps this one should go there too. I am just not sure if I should keep it there under my username or perhaps create fastai/git-tools? I’d feel odd to have tools that other people wrote under my account.

What do you think?

12

Jeremy created https://github.com/fastai/git-tools, so if you don’t mind when you feel it’s ready, i.e. after finding someone tested it on windows, please submit a PR into that repo? Just toss it into the top folder to start with - we will organize it later.

13

Sure @stas. I will get it verified on Windows 7&10 and then will submit a PR. Thanks a lot for helping me through the entire process.

1 Like
14

hi @stas,
I tested with Windows 10 and Windows 7. It works like charm. I have requested for PR at fastai/git-tools as discussed. PR Link: Added Hub-install.py Script #1

~Gokkul

15

Awesome, it is in the repo now.

I updated the dev index to move this project to the completed projects section and added a link to your PR.

And linked to the script from https://docs.fast.ai/dev/git.html#hub

Thanks again, @gokkulnath!

16

Thanks a lot @stas. That was my first open source contribution. It feels great to get your first PR Merged. Thanks again for being patient and guiding me.

~Gokkul

17

Yay! That was an excellent start, @gokkulnath.

Looking forward to seeing your future PRs if and when you get inspired to do so.

18

One idea for your 2nd contribution, if you’re interested, would be to update the readme in that repo with something that outlines what the repo is, and has an index of scripts there, with info about each one.

Of course, at this stage, we have just one script! But we’ve got to start somewhere, and this would be a good place to document what your script does… :slight_smile:

1 Like
19

That and we have 3 scripts, these 2 in addition to hub installer:

which should be moved there too, and https://docs.fast.ai/dev/git.html updated.

and probably make a copy of fastai-make-pr-branch and turn it into non-fastai specific version github-make-pr-branch, since it works just as well with any github project. Just need to remove the hardcoded fastai username and replace it with a command line argument.

20

Of course, i am very much interested and looking forward to contribute more. I have started working on fixing the issue with jupyter lab nbstripout currently. Meanwhile i will get this one done as well.

~Gokkul

1 Like