Undoing processing from a tokenizer

I am trying to undo the process that the FastAI basic tokenizer does where it adds the special xxspecial token to the text. I have looked through the documentation and found how to add it using a tokenizer, but not how to undo it. Is there any particular method I can use in FastAI to accomplish this?

Thank you for your time and consideration.

1 Like