This would be a great discussion for the next walk-thru session.
My intial thoughts are this is exactly why we need to have good “tests” that allow one to easily exercise a codebase and find issues really quickly - I love nbdev for this as I can just run a whole bunch of notebooks and find error super quickly though gitlab ci (or github actions if that’s your thing).
Looking forward to the discussion!
2 Likes
The git diff in github is really bad usually for jupyter notebooks. Do any of you have suggestions to get better context of things we work in github itself.
For reviewing PRs, I know there is review-notebook-app which I have seen @wgpubs using
2 Likes
There’s nothing in GitHub itself (other than ReviewNB, which is great for PRs), but inside jupyter lab/notebook, nbdime is very nice.
3 Likes
That’s why you have a raft of unit tests to run and find if anything breaks every time you upgrade the dependencies. Couple that with GitHub and version control you create a release strategy
The internal design of notebooks leads to default propensity to mess up git repositories, histories and diffs indeed. Contributors need sustained attention in their commits. I’ve come to use two packages to work around this and helped with my “acceptance therapy”. jupytext and nbstripout are available via conda-forge or pip. Hope this helps; they may have been already mentioned by Jeremy, now that I think of it.
1 Like
I might be biased, but I prefer to use nbdev 
4 Likes
CHMOD:
Here follows an extract of the man page on mac for chmod note you can also:
Users-Mac-Pro:clean mac$ ls -l chmod.txt
-rw-r--r-- 1 mac staff 18277 1 Jun 09:00 chmod.txt
Users-Mac-Pro:clean mac$ chmod 777 chmod.txt
Users-Mac-Pro:clean mac$ ls -l chmod.txt
-rwxrwxrwx 1 mac staff 18277 1 Jun 09:00 chmod.txt
Users-Mac-Pro:clean mac$ chmod 600 chmod.txt
Users-Mac-Pro:clean mac$ ls -l chmod.txt
-rw------- 1 mac staff 18277 1 Jun 09:00 chmod.txt
Users-Mac-Pro:clean mac$ chmod 400 chmod.txt
Users-Mac-Pro:clean mac$ ls -l chmod.txt
-r-------- 1 mac staff 18277 1 Jun 09:00 chmod.txt
MODES
Modes may be absolute or symbolic. An absolute mode is an octal number
constructed from the sum of one or more of the following values:
4000 (the set-user-ID-on-execution bit) Executable files with
this bit set will run with effective uid set to the uid of
the file owner. Directories with the set-user-id bit set
will force all files and sub-directories created in them to
be owned by the directory owner and not by the uid of the
creating process, if the underlying file system supports
this feature: see chmod(2) and the suiddir option to
mount(8).
2000 (the set-group-ID-on-execution bit) Executable files with
this bit set will run with effective gid set to the gid of
the file owner.
1000 (the sticky bit) See chmod(2) and sticky(8).
0400 Allow read by owner.
0200 Allow write by owner.
0100 For files, allow execution by owner. For directories,
allow the owner to search in the directory.
0040 Allow read by group members.
0020 Allow write by group members.
0010 For files, allow execution by group members. For directo-
ries, allow group members to search in the directory.
0004 Allow read by others.
0002 Allow write by others.
0001 For files, allow execution by others. For directories
allow others to search in the directory.
For example, the absolute mode that permits read, write and execute by
the owner, read and execute by group members, read and execute by others,
and no set-uid or set-gid behaviour is 755 (400+200+100+040+010+004+001).
The symbolic mode is described by the following grammar:
mode ::= clause [, clause ...]
clause ::= [who ...] [action ...] action
action ::= op [perm ...]
who ::= a | u | g | o
op ::= + | - | =
perm ::= r | s | t | w | x | X | u | g | o
The who symbols ``u'', ``g'', and ``o'' specify the user, group, and
other parts of the mode bits, respectively. The who symbol ``a'' is
equivalent to ``ugo''.
The perm symbols represent the portions of the mode bits as follows:
r The read bits.
s The set-user-ID-on-execution and set-group-ID-on-execution
bits.
t The sticky bit.
w The write bits.
x The execute/search bits.
X The execute/search bits if the file is a directory or any
of the execute/search bits are set in the original (unmodi-
fied) mode. Operations with the perm symbol ``X'' are only
meaningful in conjunction with the op symbol ``+'', and are
ignored in all other cases.
u The user permission bits in the original mode of the file.
g The group permission bits in the original mode of the file.
o The other permission bits in the original mode of the file.
The op symbols represent the operation performed, as follows:
+ If no value is supplied for perm, the ``+'' operation has no
effect. If no value is supplied for who, each permission bit spec-
ified in perm, for which the corresponding bit in the file mode
creation mask is clear, is set. Otherwise, the mode bits repre-
sented by the specified who and perm values are set.
- If no value is supplied for perm, the ``-'' operation has no
effect. If no value is supplied for who, each permission bit spec-
ified in perm, for which the corresponding bit in the file mode
creation mask is clear, is cleared. Otherwise, the mode bits rep-
resented by the specified who and perm values are cleared.
= The mode bits specified by the who value are cleared, or, if no who
value is specified, the owner, group and other mode bits are
cleared. Then, if no value is supplied for who, each permission
bit specified in perm, for which the corresponding bit in the file
mode creation mask is clear, is set. Otherwise, the mode bits rep-
resented by the specified who and perm values are set.
Each clause specifies one or more operations to be performed on the mode
bits, and each operation is applied to the mode bits in the order speci-
fied.
Operations upon the other permissions only (specified by the symbol ``o''
by itself), in combination with the perm symbols ``s'' or ``t'', are
ignored.
EXAMPLES OF VALID MODES
644 make a file readable by anyone and writable by the owner
only.
go-w deny write permission to group and others.
=rw,+X set the read and write permissions to the usual defaults,
but retain any execute permissions that are currently set.
+X make a directory or file searchable/executable by everyone
if it is already searchable/executable by anyone.
755
u=rwx,go=rx
u=rwx,go=u-w make a file readable/executable by everyone and writable by
the owner only.
go= clear all mode bits for group and others.
g=u-w set the group bits equal to the user bits, but clear the
group write bit.
As I said before man pages and info are your friends on any unix system be it ubuntu, macosx cygwin sun dec
1 Like
That’s a good reminder I should reassess my modus operandi indeed
I’ve been giving some thought to last night’s session, and the use of symbolic links to create persistence for dependency updates. I am a much less sophisticated user of Linux than (I suspect) anyone else here, so I want to give a perspective as a relative novice. I have a couple of concerns, and fully understand that they may not apply to more sophisticated users but maybe one day someone will be watching these videos, recognize themselves in this message, and feel a little less alone. 
-
The changes are buried deep in the shell. If you encounter bugs or conflicts a year from now, you need to remember what you did previoulsy to try to fix it. For Jeremy and others this may pose no problem because you have a deep understanding of the Linux paths and how they interact. For me, doing what Jeremy did is more like following a recipe. I get the gist, but would have a hard time troubleshooting problems. I could see this very clearly when he was troubleshooting why Python did not seem to be working. I lack the knowledge to troubleshoot in this way.
-
I have concerns about using pip and mamba/conda together. Jeremy’s initial advice was to not do this, and it is the advice I’ve read elsewhere. Last night, he commented that some dependencies can be downloaded with pip into a mamba environment without problem. I believe that, but I don’t think I’d know which are okay and which are not. And I don’t think I’d be able to fix conflicts if they arose. Also, does
environment.yamlandrequirements.txtstill work if you mix package managers? I’m not sure, but it illustrates the sort of “hidden issues” that I’m worried will arise from these sorts of bespoke solutions.
I understand why Jeremy does installations this way. It allows him to boot up Paperspace and get to work quickly. As a less experienced user, I put more of a premium on things working, since I am not confident that I can fix them if they are broken. I’m willing to sacrifice a bit of efficiency, especially since this is not a full time gig for me.
My proposed solution (for simpletons such as myself) is to create a notebook (call it “init” or something) that installs all of the updates and dependencies that I want to be persistent. This would allow me to use mamba instead of pip. I could run this notebook every time I start. It’s a bit inefficient and not as elegant as the solution Jeremy demonstrated, but it is also more transparent and seems less likely to break things in a way that I can’t fix.
I’d be interested in thoughts about and criticisms of this idea.
SSH key persistence may be an exception. It could probably be done in a notebook with a bash shell, but it’s more of a set it and forget it thing, so may be best done as Jeremy illustrated.
1 Like
I find using the 3 numbers more intuitive than remembering the letter codes (u,w,g etc.) For me the permissions are just a set of 3 groups with 3 switches in each. Each set contains (r)ead , (w)rite, e(x)ecute switches. When I turn a switch on, it turns that permission on. And which group it’s in determines who gets that. If that switch is turned on in the “user panel”, it means it affects me. If it is in the “group” set (the middle one) then it affects the permissions of the group that I may or may not belong to. if it’s in the 3rd set (the right most) then everyone gets to have that option.
I literally imagine them to be 9 switches with nine bulbs with rwx rwx rwx written under them. When I turn the switches on the bulbs turn on. Then I just read the turned on bulbs and convert them from binary to octal (which actually looks like decimal so it doesn’t hurt to think that)
and then I have the number that I need to give to my chmod command.
After a while it doesn’t seem such a long winded process and I can always reason through it and figure out what the perms should be based on what I want (and which switches should be turned on) which are really just bits in the Unix file permissions bit-mask.
So 777 means all switches are on and it translates to 111 111 111 or rwx rwx rwx .
4 Likes
I have struggled with this, and my solution has been a pretty low tech one, which is to just leave a readme.txt file in a given directory to remind me what the heck I was doing. It’s too bad that Unix with thousands of utilities does not have a “logbook” utilitiy where one can just quickly record the actions they’re taking. But I get your point, sometimes these changes if not recorded can be cause for confusion after some time has elapsed.
1 Like
I’m the same and that’s what I do too, but I felt like jumping into binary bit flags was a bit too deep to go at this early stage so I went with the method that avoided that!
3 Likes
I’ve decided to stop streaming walkthrus on YouTube Live, since there’s not many folks using it, and it’s a bit of a pain to set up.
I’ve also set up a persistent Zoom URL, so hopefully the link will be the same for every session now.
3 Likes
Can we touch on nbdime diff method for ipynb files, some of the modules you use from nbextensions for various extracts to .Py files you showed once. Nbdev seems like is for contributing packages to opensource with documentation, test files simplified from notebooks. And then there is fastpages for blogging. This whole space is a bit overwhelming set of capabilities. Perhaps the purpose, use case when to use which of these modules simplified much appreciated.
1 Like
For sure - we’re not quite up to that yet but we’ll certainly get there! I’ll add it to the top post
Persistent storage and .bash.local on Paperspace - walkthru 3 (Jun 1):
- Restarting a machine after the session the fastcore version was back to 1.4.2
- The .local folder was still in /storage/ but sym linking with
ln -s /storage/.localgave me this error:ln: failed to create symbolic link './.local': File exists - Running
rm -rf .localand thenln -s /storage/.localworked and the fastcore version was back to 1.4.3 - Redoing symlinks manually each time is obviously not preferred. So a .bash script is maybe required.
Questions
- For a complete walkthru in Paperspace can we start with setting up our own bash initialisation scripts from scratch?
- What are the key steps to persisting files in Paperspace? Creating a bash script is obviously one.
- What are the benefits of doing so versus importing at runtime?
- I saved my ssh keys to the .ssh folder and was able to connect to github and clone my repo. How to persist these keys?
- Paperspace provides a way to link to your Github account - is this only for https connections?
The bash script is touched on at 55:20 minutes into the session and the script does indicate removing .local and many other folders before symlinking to the persisted files. But we didn’t walk through setting this up and then went on to slightly other topics like git and ssh keys.
2 Likes
Will you still be recording it? It’s not possible for me to attend live at 03.00 AM local time, so I watch them a few hours later. Would be much appreciated.
Yup still recording it.
2 Likes
Yes, let’s do that today.
4 Likes
How about we instead today learn how to create a python script that does that? Then we can still automate it (if you want to), but can develop it in python. We can set that up in today’s session.
I strongly recommend not using environment.yml or requirements.txt if you can avoid it. Sometimes it’ll be needed because you need to use some old software which isn’t up to date with current versions of dependencies, in which case you should create a new conda environment for it.
Now I think about it, I can’t recall ever actually seeing a fast.ai student have any issue come up when using both pip and conda together. I don’t think I’ve faced one myself. So perhaps we’re over-complicating things by worrying about this!..
6 Likes