Package com.londogard.nlp.tokenizer

Types

public final class CharTokenizer implements Tokenizer

A Character Tokenizer which returns a token for each character in the string.

Wraps a HuggingFace tokenizer. These are subword tokenizers meaning they return subword tokens. Instantiated by calling the HuggingFace models name.

A SentencePiece Tokenizer. This is a subword-tokenizer meaning that it return subword-tokens, e.g. "hey" might end up "h", "ey".

public final class SimpleTokenizer implements Tokenizer

A simple tokenizer which allows you to define your own whitespace to split upon.

public interface Tokenizer

Tokenize a string into multiple tokens

Special tokens that is usable to Machine Learning.

public enum VocabSize extends Enum<VocabSize>