Live coding 11

This topic is for discussion of the 11th live coding session

<<< session 10session 12 >>>

Links from the walk-thru

What was covered

  • (please contribute here)

Video timeline by @daniel and @mattr

00:00 - Recap on Paddy Competition
04:30 - Tips on getting votes for Kaggle notebooks
07:30 - Gist uploading question
10:30 - Weights and Biases Sweep
14:40 - Tracking GPU metrics
16:40 - fastgpu
20:00 - Using gitconfig
21:00 - Analysis notebook
26:00 - Parallel coordinates chart on wandb
31:30 - Brute force hyperparameter optimisation vs human approach
37:30 - Learning rate finder
40:00 - Debugging port issues with ps
42:00 - Background sessions in tmux
46:20 - Strategy for iterating between notebooks
49:00 - Cell All Output toggle for overview
50:50 - Final transform for vit models
52:05 - swinv2 fixed resolution models
53:00 - Building an ensemble - appending predictions
55:50 - Model stacking
57:00 - Keeping track of submission notebooks


would it be possible to use something like linear regression to figure out weights for model ensemble ?

1 Like

kaggle is another system for ‘experiment tracking’ in a way. when you save and run all (commit) it will save notebook with outputs, input data, output data under a version, so you can come back and recheck

btw, just noticed when you run out of gpu quota on kaggle and have like minutes left in your weekly quota, you can still start the save and run all and get additional time (up to 9 hours - kernel run time) :slight_smile: .

1 Like

Congratulations to @radek! First gold medal as a notebooks contributor. :tada: :tada: :tada:

Thank you for all your contribution to the community. I learn a lot from you. :slight_smile:


Regarding finding out about processes that are holding a port (say, 8889), I use this:

On Linux:

netstat -ano | grep 8889

On macOS, there is no -o option, but -v can be used. We’re trying to get it to list the port numbers.

netstat -van | grep 8889

If you don’t know the port you can also grep for a substring in the process name to filter out just the lines containing that substring. (e.g., grep python or grep -i jupyter note that the -i option makes the grep case insensitive.)

P.S. On a paperspace instance you’ll need to install netstat first. On some systems you may need to use sudo depending on how locked down the network binaries are on a given system :smiley:

Oh wow, Sarada, this is wonderful! :partying_face: My first ever Kaggle Notebook gold! :1st_place_medal: Earned with the help of our amazing community! :heart: It couldn’t get better than this! :smile:

Thank you so very much for making this happen and my heartfelt thank you to everyone who contributed their votes :slightly_smiling_face:

And thank you so much for your very kind words! I continue to learn a lot from everyone here. It was wonderful to see you today in the walk-through today, Sarada!!! :slightly_smiling_face:

BTW what surprises me quite a bit is the number of medals in discussions. I can barely remember talking online that much :smile: But oh well, if there is a record to prove it, I guess it must have happened :slight_smile:

Thank you so very much again everyone for this wonderful surprise! :slight_smile: :heart:


Something peculiar I noticed on Paperspace today is that I was getting a very long error when creating the data loader with the same exact code that worked on a different machine yesterday.

I tried restarting the kernel but got the same error. Then I stopped the machine (P5000) and started a new RTX4000 machine and ran the same exact notebook and the same code ran without error. No idea why but seems there a problem somewhere.

1 Like

i had the same error… by reducing the size (in a hunch) of the aug_transforms, that error went away….(both on P5000)


There are 19 people on the same score as me in the Paddy competition now. Is the order arbitrary or in order of submission when scores are the same?

1 Like

The order looks random to me. You are doing a good job. :slight_smile:

1 Like

I always thought it’s ordered by submission time if score is same …

The time here could be of the last submission which not necessarily is the best one i think…
Edit: i meant that if you submit a solution kaggle records a time of it, if it’s your the best scoe, the score and hopefully position gets updated on leaderboard. However if score is lower than best, only submission time is recorded.

1 Like

Ok, I had the same on my local computer. Could you check my full trace and compare it to yours?
Mine disappears when I run the same line again. It is weird.

Is there are generally accepted description for the type of multiple target model you started to build today Jeremy? It’s not multi-modal because we are only using images for training. Would this be correctly referred to as a multi-head model?

1 Like

I’m not sure I’ve really heard them being given a name before! That description you linked to seems close enough, however.

Although as you’ll see tomorrow I’ve come up with a much simpler approach…


How can we get the sweep ids from the wandb experimentation? I’m running to generate the data for the as Jeremy did. I don’t know how to construct the sweep id for creating dictionaries for the dataframe? Any hint? Thanks.

I really like using the lsof command for this category of issues.

lsof -i :4321 # gets the process that "opened" the port number 4321

Pretty handy with hunting down (process that) occupied local ports, locked/opened files etc., lsof (list open files) does a lot more & is included with most distros.


Adding the blogpost mentioned by Jeremy about the experiments on the topic 31:30 - Brute force hyperparameter optimisation vs human approach.


Ohh, I can help you with that on the live session on Friday 24. I updated the Readme to make it more clear.

That would be great. Would you please share the link to the live session on 24th. Thanks.