I think its best if you use the data block API as it offers more flexibility and I would think that it would become the gold standard for data processing as the library matures. I’ve successfully used it in a custom dataset for LM. Here is my documentation for that.
1 Like