CharTokenizer

public final class CharTokenizer implements Tokenizer

A Character Tokenizer which returns a token for each character in the string.

Constructors

A more efficient approach for native tokenizers, i.e. HuggingFaceTokenizer