Just wondering if you have any guidelines on when to use bash vs python. For example one can setup folder structure for lesson 1 homework using bash or python. I found that “!” commands are independent from each other and certain tasks are more readable in python.
I think the answer is “whatever’s easiest for you”. I use all of the above, depending on the task.
I often just use an emacs macro for doing things, other times I use the find command, and still others I’ll actually use pytohn commands. I suspect it just depends on what you’re comfortable with and know the best.
I think it’s good to have everything necessary to reproduce your analysis results in one place - so I like to have all the stuff for creating the validation and sample sets in the same notebook as the analysis.
If you need a more complex script, I’d suggest calling it from the notebook, so you still have a reproducible set of steps.
Reproducibility of analysis is something that I learned in my time competing on kaggle, and talking to the top competitors, is one of the most important things in getting good results.