nlp/com.londogard.nlp.tokenizer/HuggingFaceTokenizerWrapper

HuggingFaceTokenizerWrapper

public final class HuggingFaceTokenizerWrapper implements Tokenizer

Wraps a HuggingFace tokenizer. These are subword tokenizers meaning they return subword tokens. Instantiated by calling the HuggingFace models name.

Constructors

HuggingFaceTokenizerWrapper

public HuggingFaceTokenizerWrapper HuggingFaceTokenizerWrapper(String modelName)

HuggingFaceTokenizerWrapper

public HuggingFaceTokenizerWrapper HuggingFaceTokenizerWrapper(HuggingFaceTokenizer tokenizer)

Functions

public final Array<Encoding> batchEncode(List<String> texts)

public List<List<String>> batchSplit(List<String> texts)

A more efficient approach for native tokenizers, i.e. HuggingFaceTokenizer

public final Encoding encode(String text)

public final HuggingFaceTokenizer getTokenizer()

public List<String> split(String text)

Properties

private final HuggingFaceTokenizer tokenizer