nlp/com.londogard.nlp.tokenizer/Tokenizer

Tokenizer

public interface Tokenizer

Tokenize a string into multiple tokens

Functions

public List<List<String>> batchSplit(List<String> texts)

A more efficient approach for native tokenizers, i.e. HuggingFaceTokenizer

public abstract List<String> split(String text)

Inheritors

HuggingFaceTokenizerWrapper

SentencePieceTokenizer

SimpleTokenizer

SimpleSentenceTokenizer