I want to design a custom tokenizer module in Python that lets users specify what tokenizer(s) to use for the input. For instance, consider the following input: Q: What is a good way to achieve this? A: I am not so sure. I think I will use Python. I want to be able to provide […]
