Interface ITokenizer

Interface for tokenizer

Namespace: DotnetPrompt.Abstractions.Tools
Assembly: DotnetPrompt.Abstractions.dll
Syntax
public interface ITokenizer
Remarks

A tokenizer is a software component that breaks down a piece of text into smaller units called tokens.

Methods

| Improve this Doc View Source

Encode(string)

Encode text as a list of numerical tokens

Declaration
IList<int> Encode(string text)
Parameters
Type Name Description
string text
Returns
Type Description
IList<int>