Fastai-nbstripout: stripping notebook outputs and metadata for git storage

stas · February 24, 2019, 7:27am

Any luck with this version of the same?

python -c 'import fastai.utils.collect_env; fastai.utils.collect_env.show_install(1)'

And the manual command worked fine… Does this mean I have to do everything manually? How will I create a new branch?

No, we just need to sort out what the problem is.

First, you can disable the filter, until we sort it out:

python tools\trust-origin-git-config -d

and only apply the stripout to ipynb files before committing if you plan to contribute to the notebooks. So it’s no problem - just not automated.

Now, someone else would be probably more qualified to help you out there, since I don’t know windows. So unless this happens please bear with me. Going back to the git filters that you have a problem with. This:

python tools\trust-origin-git-config -e

generates .gitconfig with the instructions to git how to run the stripout filter.

In the past we had issues with this setup on windows, it had to do with the number of \ we had to put in - madness! I thought it was figured out. I guess it is not.

So if the manual invocation of the script works, it’s all about figuring out why the autogenerated .gitconfig isn’t working for you. So have a look inside, perhaps tweak it and see if you can find if it’s the number of \ in the path that’s the problem. Notice that python tools\trust-origin-git-config -e enables the filters and -d disables them.

ilovescience · February 25, 2019, 6:58am

I replaced all the backslashes to a single forward slash and it worked and the git checkout commands work and now I cannot replicate the original error when changing back to 4 backslashes! So I will continue the git Notes tutorial and see how it goes, and let you know if the error returns.

Also, fastai.utils command now works. I had missed a folder that needed to be added to my PATH. When I run the command I get the following:

=== Software ===
python        : 3.7.1
fastai        : 1.0.45
fastprogress  : 0.1.19
torch         : 1.0.0
torch cuda    : 9.0 / is available
torch cudnn   : 7005 / is enabled

=== Hardware ===
torch devices : 1
  - gpu0      : GeForce GTX 1050 Ti

=== Environment ===
platform      : Windows-10-10.0.17763-SP0
conda env     : base
python        : D:\Anaconda3\python.exe
sys.path      :
D:\Anaconda3\python37.zip
D:\Anaconda3\DLLs
D:\Anaconda3\lib
D:\Anaconda3
D:\Anaconda3\lib\site-packages
D:\Anaconda3\lib\site-packages\win32
D:\Anaconda3\lib\site-packages\win32\lib
D:\Anaconda3\lib\site-packages\Pythonwin
no nvidia-smi is found

stas · February 25, 2019, 7:14am

Good. So I suppose you have a real bash on windows. I think whoever we made it work for last on windows, had no bash, just the command prompt. How do we tell bash env on windows - what is your SHELL environment variable - If it’s on windows? On linux it is usually set to /bin/bash.

and then can you adjust tools\trust-origin-git-config so that it generates the format that you found working, it’s most likely just this part:

is_windows = hasattr(sys, 'getwindowsversion')
cmd = "tools/fastai-nbstripout" if not is_windows else r"python tools\\\\fastai-nbstripout"

So it probably needs to be changed to:

is_windows = hasattr(sys, 'getwindowsversion')
if SHELL is in os.environ: is_windows = False
cmd = "tools/fastai-nbstripout" if not is_windows else r"python tools\\\\fastai-nbstripout"

so we pretend we are on unix if it’s bash. Except, please figure out what that condition should be so that it works for you. This is untested.

Will probably make it then into is_bash_like flag, so it’ll be more intuitive.

Thanks.

edit:

I searched on SO a bit, so this might be all you need to do the right conditional:

echo $BASH # should be set for BASH
What type of bash:
https://stackoverflow.com/a/33828925/9201239
Bash sets the shell variable OSTYPE. From man bash:

Automatically set to a string that describes the operating system on which bash is executing.

case "$OSTYPE" in
  linux*)   echo "Linux / WSL" ;;
  darwin*)  echo "Mac OS" ;; 
  win*)     echo "Windows" ;;
  msys*)    echo "MSYS / MinGW / Git Bash" ;;
  cygwin*)  echo "Cygwin" ;;
  bsd*)     echo "BSD" ;;
  solaris*) echo "Solaris" ;;
  *)        echo "unknown: $OSTYPE" ;;
esac

So you can probably get the logic right from BASH and OSTYPE env vars.

And I think we are dealing multiple types of shells on windows:

Windows Bash (/ sep)
Git Bash / or \ sep?
not Bash /(\ sep)
Cygwin bash

I guess yours is the first one.

partho · March 5, 2019, 10:37am

[Resurrecting this thread on @stas request]

hi @ilovescience,

I have been able to clone (with tools/run-after-git-clone) and create a new branch. See the pictures below.

Can you please pull down the latest source and see if you are getting any errors?
If so - can you please tell me (a) the errors and (b) the exact steps that led to the errors.

Happy to ‘sit’ with you till this is resolved. Win10 is my primary environment for DL as well.

On cmd.exe

image.png1868×472 40.6 KB

On ubuntu.exe

image.png1838×377 40.5 KB

stas · March 5, 2019, 4:17pm

I guess the main question - does Bash on win10 behaves likes unix (uses unix / paths or not). It’s really just figuring out whether is_windows in tools\trust-origin-git-config (see my last comment) should be True or bash/win10 or not.

ilovescience · March 6, 2019, 8:48pm

Hello Partho,

Thanks for the response… I have not had much time to get back to working on this, but I was using Git Bash instead of the Command Prompt so that could have been the reason why the program was confused, because I think Git Bash behaves like Unix but Command Prompt does not…

When I get a chance to work on this again, I will definitely let you know if there are any errors with this aspect and with the development process.

partho · March 8, 2019, 3:18am

@stas w.r.t. testing is_windows here is what i find:

environment	is_windows	“/” or “\”?	comments
cmd.exe/powershell	true	“\”	Native windows command prompts.
bash in windows subsystem for linux	false	“/”	WLS is a true emulation layer. executables inside are true linux executables running in a true linux environment
bash in mingw64	true	?	i dont understand the how this works. it looks like a simulation layer for linux binaries. i am very curious to know its usage given wsl super hi-fidelity emulation of linux

In short I believe the trust-origin-git-config script is detecting the environment accurately.

I’ll make the necessary fixes if there are reports to the contrary.

stas · March 8, 2019, 3:25am

Looks good, @partho.

But I’m not sure how you made the entry for “bash in windows” to be is_windows == False
It’s derived from:
is_windows = hasattr(sys, 'getwindowsversion')
so probably it should be fixed to be false in that case, no?

or are you saying under bash/windows hasattr(sys, 'getwindowsversion') returns False? It pretends to be unix?

partho · March 8, 2019, 4:01am

are you saying under bash/windows hasattr(sys, 'getwindowsversion') returns False? It pretends to be unix?

Yes it returns false & yes it ‘pretends’ to be linux**.

If I understand your question correctly, you are asking about WSL.
Bash (and all other processes) when run under WSL get to see a linux environment (the OS APIs, filesystem etc.). They are binary identical to ones found on linux. For most purposes they don’t know that the linux environment is actually being emulated inside an windows OS. Inception movie type thingy going on in there.

Of course that emulation layer does not have 100% fidelity yet. The biggest missing piece relevant to us is CPU/CUDA access.

**linux here stands for all distros of linuix supported by WLS.

stas · March 8, 2019, 4:26am

Thank you for the detailed explanation, @partho!

What about git bash on windows? I understand that there is that variation too.

partho · March 8, 2019, 5:05am

That was the 3rd row in the matrix above

environment is_windows “/” or “\”? comments

bash in mingw64 true ? i dont understand the how this works. it looks like a simulation layer for linux binaries. i am very curious to know its usage given wsl super hi-fidelity emulation of linux

I don’t have much experience with this one and I am not sure how well this works compared to the other 2. Given we have WSL, I would be very surprised if any serious work is done through this.

Do you want this to a be a supported environment as well?

stas · March 8, 2019, 5:12am

Ah, that’s git bash. Understood. Thank you.

Probably not until we get someone wanting it, perhaps it just would never happen so why waste your time.