nlp/com.londogard.nlp.tokenizer/SentencePieceTokenizer

SentencePieceTokenizer

public final class SentencePieceTokenizer implements Tokenizer

A SentencePiece Tokenizer. This is a subword-tokenizer meaning that it return subword-tokens, e.g. "hey" might end up "h", "ey".

Constructors

SentencePieceTokenizer

public SentencePieceTokenizer SentencePieceTokenizer(Path modelPath, Path vocabPath)

Types

public class Companion

Functions

public List<List<String>> batchSplit(List<String> texts)

A more efficient approach for native tokenizers, i.e. HuggingFaceTokenizer

public final Set<String> getVocab()

public List<String> split(String text)

Properties

private final Set<String> vocab