nlp/com.londogard.nlp.tokenizer/SimpleTokenizer

SimpleTokenizer

public final class SimpleTokenizer implements Tokenizer

A simple tokenizer which allows you to define your own whitespace to split upon.

Constructors

SimpleTokenizer

public SimpleTokenizer SimpleTokenizer(Boolean splitContraction, String whitespaceRegex)

Types

public class Companion

Functions

public List<List<String>> batchSplit(List<String> texts)

A more efficient approach for native tokenizers, i.e. HuggingFaceTokenizer

public List<String> split(String text)