Just wondering if you have any guidelines on when to use bash vs python. For example one can setup folder structure for lesson 1 homework using bash or python. I found that “!” commands are independent from each other and certain tasks are more readable in python.
Any preferences based on experience?
I think the answer is “whatever’s easiest for you”. I use all of the above, depending on the task.
I often just use an emacs macro for doing things, other times I use the find command, and still others I’ll actually use pytohn commands. I suspect it just depends on what you’re comfortable with and know the best.
I think it’s good to have everything necessary to reproduce your analysis results in one place - so I like to have all the stuff for creating the validation and sample sets in the same notebook as the analysis.
If you need a more complex script, I’d suggest calling it from the notebook, so you still have a reproducible set of steps.
Reproducibility of analysis is something that I learned in my time competing on kaggle, and talking to the top competitors, is one of the most important things in getting good results.
Anything you can do in bash you can do in python, although usually in a more verbose manner.
If you want to automate bash calls from within a python script, check out the subprocess module, particularly check_call, check_output and Popen.