nlp/com.londogard.nlp.tokenizer/Tokenizer/batchSplit

batchSplit

public List<List<String>> batchSplit(List<String> texts)
Content copied to clipboard

A more efficient approach for native tokenizers, i.e. HuggingFaceTokenizer