Thanks again @henripal for the time and effort you are putting into this competition! These are, again, extremely usegul findings.
I would propose the following:
- We need to make sure participants use the right Train/ Test split using the correct UCRArchive_2018.
- I think we should propose a new rule in competition (because it’s useful in real problems). To perform any hyperparameter search, we should use a validation set out of the training set, and then report the test set performance. That’s best practice.
To achieve both points we should update the notebook we created to prepare data. I cannot work on that until tomorrow. If anybody can do it before that’d be great. Otherwise I’ll do it tomorrow.
I’d propose we continue the discusion on how to proceed with the TS project in the TSSD study group since its beyond the scope of this competition. All those interested are welcome to participate.