SOURCE CODE: Mid-Level API

I’ve been swimming in the code for months… learn something new about this library every day… that is so easy, cool, and awesome. Thank you nbdev :wink:

1 Like

Well, another easy thing to do is just search for the function name in the github repository of the library itself! (select search “in this repository”). You will get bunch of results but you know which to chose from.

3 Likes

Wasn’t sure if that would work on private functions, but if it’s just searching the raw code then makes sense

Yes! doc does the work 90% of the time, but when in doubt I default to good old ?? then vim into the actual code.

3 Likes

It’s just a raw text match.

1 Like

I maintain a clone of the main repos (Fastai2 and fastcore) on my laptop and always have Sublime Text open with these 2 folders. Any function I need to find is then a jump away. I find jumping between notebooks to generally be quite slow as a proper universal searching tool for all nbs in a folder is still not available. (Yes, source_link works buts it’s still kind of slow). Jumping to functions using something like VIM or an IDE is generally much more convenient and much faster.

Note that there are some important things you’d find only in Fastcore (like Transform) so you should also search there sometimes before scratching your head for an hour :sweat_smile:

Alright first blog is officially up folks :slight_smile:

Though just realized this should say Top Level API

3 Likes

We saw it even before you shared it here :wink:

1 Like

But this time I messed with CSS (shutters) :wink:

I meant the published version on fastblog. :slight_smile:

1 Like

Oh geez, you were watching my drafts that’s terrifying :wink:

1 Like

WAtching every step now. Not just me, @init_27 and @barnacl too! :wink:

2 Likes

This is great!

We’d decided that all of us would have something by Sunday with respect to the PETs Dataset. That was my goal initially too. But then as I started looking at the low level APIs (Transform, Pipeline, TfmdDL, Dataloader, TypeDispatch, etc.) I got carried away and just started putting time into understanding those. So far I’ve almost completely understood every line of code in Dataloader (with comments) and Transform. Pipeline and TfmdDL is next on the list, and then TypeDispatch. I thought I’ll do the Datablocks API for the purpose of PETs but then I didn’t want to lose the flow.

Dataloder has a lot of clever code that needs time to get through. There’s a bit or two that I didn’t (entirely) understand because they imported things from the PyTorch library which had no documentation. But nonetheless, I’ve managed to make sense of every line of code in it.

I think I’ll currently keep going in this direction for now. :sweat_smile:

2 Likes

@arora_aman fell into the same trap :wink: (which is a great trap, both in the rabbit hole and the knowledge gained).

Very impressive!

For the last, it was easier for me to see it and understand it via the vision transforms specifically

Yup. That’s kind of what my goal is too!

1 Like

Also, one bit on this is I may step on your toes a little bit. I plan on doing an article on dblock.summary() (probably like an article and a half from now?) because it’s so powerful, and I’ll be introducing how to use it to debug your medium level stuff as well, hope that’s okay :slight_smile:

1 Like

It’s fine @muellerz. I’m not really worried all that much about who’d write about what. All I really want is to understand as much of the library as possible, and in the process possibly write what hasn’t been written about. It doesn’t matter if I don’t get to write about some particular aspect of it. :sweat_smile:

2 Likes

Great, I’m will surely read this. I have a suggestion: the font size is bit smaller and makes it harder to read the text on a laptop screen (but looks great on mobile), could you make it a bit larger?.

how do you all dig through the code? in notebooks and ?? each method? My first idea was to use VSCode and use the debugger / breakpoints, but I can’t get fastai to work on my MacBook. I think it would be way easier to set breakpoints and step through the code instead of doing it manually in the notebooks.

any advice on how to explore the code?

I personally use VS code on my MacBook only which I found to be the best way of all. Maybe vim is a better option but too much of a learning curve and thus I stuck to VS code.

Most of the tips are by following Jeremy’s advice itself: https://youtu.be/Z0ssNAbe81M?t=3809

In summary, here are the shortcuts most useful:

  • Search: cmd + t
  • Navigate to sub-command: f12
  • Returning back to where you originally were: ctrl + - (or ctrl + shift + - for going back forward)

Here’s a related topic on the forum as well: Examining source code with Visual Studio

3 Likes