nlp/com.londogard.nlp.tokenizer.sentence/SimpleSentenceTokenizer

SimpleSentenceTokenizer

public final class SimpleSentenceTokenizer implements Tokenizer

A sentence tokenizer which returns each sentence as a token using simple heuristics.

Constructors

SimpleSentenceTokenizer

public SimpleSentenceTokenizer SimpleSentenceTokenizer()

Types

public class Companion

Functions

public List<List<String>> batchSplit(List<String> texts)

A more efficient approach for native tokenizers, i.e. HuggingFaceTokenizer

public List<String> split(String text)