ClassDefaultTokensCounter
Class
Provides functionality to count tokens in a given input, such as words or other units of text.
Definition
Namespace:Telerik.Documents.AI.RAG
Assembly:Telerik.Documents.AI.RAG.dll
Syntax:
cs-api-definition
public class DefaultTokensCounter : ITokensCounterInheritance: objectDefaultTokensCounter
Implements:
Constructors
DefaultTokensCounter(IEmbeddingSettings)
Initializes a new instance of the DefaultTokensCounter class with the specified embedding settings.
Declaration
cs-api-definition
public DefaultTokensCounter(IEmbeddingSettings settings)Parameters
settings
The embedding settings that define the model ID and tokenization encoding to be used.
Remarks
The settings parameter is used to configure the tokenizer encoding
based on the specified model ID and tokenization encoding. Ensure that the IEmbeddingSettings
object is properly configured before passing it to this constructor.
Methods
Count(string)
Counts the number of tokens in the specified input string.