# tokenizers Tokenization utilities * [tokenizers](#module_tokenizers) * _static_ * [.PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) * [`new PreTrainedTokenizer(tokenizerJSON, tokenizerConfig)`](#new_module_tokenizers.PreTrainedTokenizer_new) * _instance_ * [`.convert_tokens_to_ids(tokens)`](#module_tokenizers.PreTrainedTokenizer+convert_tokens_to_ids) ⇒ any * [`._call(text, [options])`](#module_tokenizers.PreTrainedTokenizer+_call) ⇒ BatchEncoding.<BatchEncodingItem.<TText, TReturnTensor>> * [`._encode_text(text)`](#module_tokenizers.PreTrainedTokenizer+_encode_text) ⇒ Array | null * [`.tokenize(text, options)`](#module_tokenizers.PreTrainedTokenizer+tokenize) ⇒ Array * [`.encode(text, options)`](#module_tokenizers.PreTrainedTokenizer+encode) ⇒ Array * [`.batch_decode(batch, decode_args)`](#module_tokenizers.PreTrainedTokenizer+batch_decode) ⇒ Array * [`.decode(token_ids, [decode_args])`](#module_tokenizers.PreTrainedTokenizer+decode) ⇒ string * [`.decode_single(token_ids, decode_args)`](#module_tokenizers.PreTrainedTokenizer+decode_single) ⇒ string * [`.get_chat_template(options)`](#module_tokenizers.PreTrainedTokenizer+get_chat_template) ⇒ string * [`.apply_chat_template(conversation, [options])`](#module_tokenizers.PreTrainedTokenizer+apply_chat_template) ⇒ ApplyChatTemplateReturn.<TTokenize, TReturnTensor, TReturnDict> * _static_ * [`.from_pretrained(pretrained_model_name_or_path, options)`](#module_tokenizers.PreTrainedTokenizer.from_pretrained) ⇒ Promise.<PreTrainedTokenizer> * [`.loadTokenizer(pretrained_model_name_or_path, options)`](#module_tokenizers.loadTokenizer) ⇒ Promise.<Array> * [`.prepareTensorForDecode(tensor)`](#module_tokenizers.prepareTensorForDecode) ⇒ Array * [`._build_translation_inputs(self, raw_inputs, tokenizer_options, generate_kwargs)`](#module_tokenizers._build_translation_inputs) ⇒ Object * _inner_ * [`~PretrainedTokenizerOptions`](#module_tokenizers..PretrainedTokenizerOptions) : [PretrainedOptions](#PretrainedOptions) * [`~TextContent`](#module_tokenizers..TextContent) : Object * [`~ImageContent`](#module_tokenizers..ImageContent) : Object * [`~MessageContent`](#module_tokenizers..MessageContent) : TextContent | ImageContent | Object * [`~Message`](#module_tokenizers..Message) : Object * [`~BatchEncodingArrayItem`](#module_tokenizers..BatchEncodingArrayItem) : any * [`~BatchEncodingItem`](#module_tokenizers..BatchEncodingItem) : any * [`~BatchEncoding`](#module_tokenizers..BatchEncoding) : Object * [`~TokenizerCallOptions`](#module_tokenizers..TokenizerCallOptions) : Object * [`~PreTrainedTokenizerCallback`](#module_tokenizers..PreTrainedTokenizerCallback) : function * [`~ApplyChatTemplateOptions`](#module_tokenizers..ApplyChatTemplateOptions) : Object * [`~ApplyChatTemplateReturn`](#module_tokenizers..ApplyChatTemplateReturn) : any * * * ## tokenizers.PreTrainedTokenizer **Kind**: static class of [tokenizers](#module_tokenizers) * [.PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) * [`new PreTrainedTokenizer(tokenizerJSON, tokenizerConfig)`](#new_module_tokenizers.PreTrainedTokenizer_new) * _instance_ * [`.convert_tokens_to_ids(tokens)`](#module_tokenizers.PreTrainedTokenizer+convert_tokens_to_ids) ⇒ any * [`._call(text, [options])`](#module_tokenizers.PreTrainedTokenizer+_call) ⇒ BatchEncoding.<BatchEncodingItem.<TText, TReturnTensor>> * [`._encode_text(text)`](#module_tokenizers.PreTrainedTokenizer+_encode_text) ⇒ Array | null * [`.tokenize(text, options)`](#module_tokenizers.PreTrainedTokenizer+tokenize) ⇒ Array * [`.encode(text, options)`](#module_tokenizers.PreTrainedTokenizer+encode) ⇒ Array * [`.batch_decode(batch, decode_args)`](#module_tokenizers.PreTrainedTokenizer+batch_decode) ⇒ Array * [`.decode(token_ids, [decode_args])`](#module_tokenizers.PreTrainedTokenizer+decode) ⇒ string * [`.decode_single(token_ids, decode_args)`](#module_tokenizers.PreTrainedTokenizer+decode_single) ⇒ string * [`.get_chat_template(options)`](#module_tokenizers.PreTrainedTokenizer+get_chat_template) ⇒ string * [`.apply_chat_template(conversation, [options])`](#module_tokenizers.PreTrainedTokenizer+apply_chat_template) ⇒ ApplyChatTemplateReturn.<TTokenize, TReturnTensor, TReturnDict> * _static_ * [`.from_pretrained(pretrained_model_name_or_path, options)`](#module_tokenizers.PreTrainedTokenizer.from_pretrained) ⇒ Promise.<PreTrainedTokenizer> * * * ### `new PreTrainedTokenizer(tokenizerJSON, tokenizerConfig)` Create a new PreTrainedTokenizer instance. ParamTypeDescription tokenizerJSONObjectThe JSON of the tokenizer. tokenizerConfigObjectThe config of the tokenizer. * * * ### `preTrainedTokenizer.convert_tokens_to_ids(tokens)` ⇒ any Converts a token string (or a sequence of tokens) into a single integer id (or a sequence of ids), using the vocabulary. **Kind**: instance method of [PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) **Returns**: any - The token id or list of token ids. ParamTypeDescription tokensTOne or several token(s) to convert to token id(s). * * * ### `preTrainedTokenizer._call(text, [options])` ⇒ BatchEncoding.<BatchEncodingItem.<TText, TReturnTensor>> Encode/tokenize the given text(s). **Kind**: instance method of [PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) **Returns**: BatchEncoding.<BatchEncodingItem.<TText, TReturnTensor>> - Object to be passed to the model. ParamTypeDescription textTTextThe text to tokenize. [options]TokenizerCallOptions.<TText, TReturnTensor>Additional tokenization options. * * * ### `preTrainedTokenizer._encode_text(text)` ⇒ Array | null Encodes a single text using the preprocessor pipeline of the tokenizer. **Kind**: instance method of [PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) **Returns**: Array | null - The encoded tokens. ParamTypeDescription textstring | nullThe text to encode. * * * ### `preTrainedTokenizer.tokenize(text, options)` ⇒ Array Converts a string into a sequence of tokens. **Kind**: instance method of [PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) **Returns**: Array - The list of tokens. ParamTypeDefaultDescription textstringThe sequence to be encoded. optionsObjectAn optional object containing the following properties: [options.pair]string | nullA second sequence to be encoded with the first. [options.add_special_tokens]booleanfalseWhether or not to add the special tokens associated with the corresponding model. * * * ### `preTrainedTokenizer.encode(text, options)` ⇒ Array Encodes a single text or a pair of texts using the model's tokenizer. **Kind**: instance method of [PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) **Returns**: Array - An array of token IDs representing the encoded text(s). ParamTypeDefaultDescription textstringThe text to encode. optionsObjectAn optional object containing the following properties: [options.text_pair]string | nullnullThe optional second text to encode. [options.add_special_tokens]booleantrueWhether or not to add the special tokens associated with the corresponding model. [options.return_token_type_ids]boolean | nullWhether to return token_type_ids. * * * ### `preTrainedTokenizer.batch_decode(batch, decode_args)` ⇒ Array Decode a batch of tokenized sequences. **Kind**: instance method of [PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) **Returns**: Array - List of decoded sequences. ParamTypeDescription batchArray | TensorList/Tensor of tokenized input sequences. decode_argsObject(Optional) Object with decoding arguments. * * * ### `preTrainedTokenizer.decode(token_ids, [decode_args])` ⇒ string Decodes a sequence of token IDs back to a string. **Kind**: instance method of [PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) **Returns**: string - The decoded string. **Throws**: - Error If `token_ids` is not a non-empty array of integers. ParamTypeDefaultDescription token_idsArray | Array | TensorList/Tensor of token IDs to decode. [decode_args]Object{} [decode_args.skip_special_tokens]booleanfalseIf true, special tokens are removed from the output string. [decode_args.clean_up_tokenization_spaces]booleantrueIf true, spaces before punctuations and abbreviated forms are removed. * * * ### `preTrainedTokenizer.decode_single(token_ids, decode_args)` ⇒ string Decode a single list of token ids to a string. **Kind**: instance method of [PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) **Returns**: string - The decoded string ParamTypeDefaultDescription token_idsArray | ArrayList of token ids to decode decode_argsObjectOptional arguments for decoding [decode_args.skip_special_tokens]booleanfalseWhether to skip special tokens during decoding [decode_args.clean_up_tokenization_spaces]boolean | nullWhether to clean up tokenization spaces during decoding. If null, the value is set to this.decoder.cleanup if it exists, falling back to this.clean_up_tokenization_spaces if it exists, falling back to true. * * * ### `preTrainedTokenizer.get_chat_template(options)` ⇒ string Retrieve the chat template string used for tokenizing chat messages. This template is used internally by the `apply_chat_template` method and can also be used externally to retrieve the model's chat template for better generation tracking. **Kind**: instance method of [PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) **Returns**: string - The chat template string. ParamTypeDefaultDescription optionsObjectAn optional object containing the following properties: [options.chat_template]string | nullnullA Jinja template or the name of a template to use for this conversion. It is usually not necessary to pass anything to this argument, as the model's template will be used by default. [options.tools]ArrayA list of tools (callable functions) that will be accessible to the model. If the template does not support function calling, this argument will have no effect. Each tool should be passed as a JSON Schema, giving the name, description and argument types for the tool. See our chat templating guide for more information. * * * ### `preTrainedTokenizer.apply_chat_template(conversation, [options])` ⇒ ApplyChatTemplateReturn.<TTokenize, TReturnTensor, TReturnDict> Converts a list of message objects with `"role"` and `"content"` keys to a list of token ids. This method is intended for use with chat models, and will read the tokenizer's chat_template attribute to determine the format and control tokens to use when converting. See [here](https://huggingface.co/docs/transformers/chat_templating) for more information. **Example:** Applying a chat template to a conversation. ```javascript import { AutoTokenizer } from "@huggingface/transformers"; const tokenizer = await AutoTokenizer.from_pretrained("Xenova/mistral-tokenizer-v1"); const chat = [ { "role": "user", "content": "Hello, how are you?" }, { "role": "assistant", "content": "I'm doing great. How can I help you today?" }, { "role": "user", "content": "I'd like to show off how chat templating works!" }, ] const text = tokenizer.apply_chat_template(chat, { tokenize: false }); // "[INST] Hello, how are you? [/INST]I'm doing great. How can I help you today? [INST] I'd like to show off how chat templating works! [/INST]" const input_ids = tokenizer.apply_chat_template(chat, { tokenize: true, return_tensor: false }); // [1, 733, 16289, 28793, 22557, 28725, 910, 460, 368, 28804, 733, 28748, 16289, 28793, 28737, 28742, 28719, 2548, 1598, 28723, 1602, 541, 315, 1316, 368, 3154, 28804, 2, 28705, 733, 16289, 28793, 315, 28742, 28715, 737, 298, 1347, 805, 910, 10706, 5752, 1077, 3791, 28808, 733, 28748, 16289, 28793] ``` **Kind**: instance method of [PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) **Returns**: ApplyChatTemplateReturn.<TTokenize, TReturnTensor, TReturnDict> - The tokenized output. ParamTypeDefaultDescription conversationArrayA list of message objects with "role" and "content" keys, representing the chat history so far. [options]ObjectAn optional object containing the following properties: [options.chat_template]string | nullnullA Jinja template to use for this conversion. If this is not passed, the model's chat template will be used instead. [options.tools]ArrayA list of tools (callable functions) that will be accessible to the model. If the template does not support function calling, this argument will have no effect. Each tool should be passed as a JSON Schema, giving the name, description and argument types for the tool. See our chat templating guide for more information. [options.documents]Array.<Record>A list of dicts representing documents that will be accessible to the model if it is performing RAG (retrieval-augmented generation). If the template does not support RAG, this argument will have no effect. We recommend that each document should be a dict containing "title" and "text" keys. Please see the RAG section of the chat templating guide for examples of passing documents with chat templates. [options.add_generation_prompt]booleanfalseWhether to end the prompt with the token(s) that indicate the start of an assistant message. This is useful when you want to generate a response from the model. Note that this argument will be passed to the chat template, and so it must be supported in the template for this argument to have any effect. [options.tokenize]TTokenizetrueWhether to tokenize the output. If false, the output will be a string. [options.padding]booleanfalseWhether to pad sequences to the maximum length. Has no effect if tokenize is false. [options.truncation]booleanfalseWhether to truncate sequences to the maximum length. Has no effect if tokenize is false. [options.max_length]number | nullMaximum length (in tokens) to use for padding or truncation. Has no effect if tokenize is false. If not specified, the tokenizer's max_length attribute will be used as a default. [options.return_tensor]TReturnTensortrueWhether to return the output as a Tensor or an Array. Has no effect if tokenize is false. [options.return_dict]TReturnDicttrueWhether to return a dictionary with named outputs. Has no effect if tokenize is false. [options.tokenizer_kwargs]Object{}Additional options to pass to the tokenizer. * * * ### `PreTrainedTokenizer.from_pretrained(pretrained_model_name_or_path, options)` ⇒ Promise.<PreTrainedTokenizer> Loads a pre-trained tokenizer from the given `pretrained_model_name_or_path`. **Kind**: static method of [PreTrainedTokenizer](#module_tokenizers.PreTrainedTokenizer) **Returns**: Promise.<PreTrainedTokenizer> - A new instance of the `PreTrainedTokenizer` class. **Throws**: - Error Throws an error if the tokenizer.json or tokenizer_config.json files are not found in the `pretrained_model_name_or_path`. ParamTypeDescription pretrained_model_name_or_pathstringThe path to the pre-trained tokenizer. optionsPretrainedTokenizerOptionsAdditional options for loading the tokenizer. * * * ## `tokenizers.loadTokenizer(pretrained_model_name_or_path, options)` ⇒ Promise.<Array> Loads a tokenizer from the specified path. **Kind**: static method of [tokenizers](#module_tokenizers) **Returns**: Promise.<Array> - A promise that resolves with information about the loaded tokenizer. ParamTypeDescription pretrained_model_name_or_pathstringThe path to the tokenizer directory. optionsPretrainedTokenizerOptionsAdditional options for loading the tokenizer. * * * ## `tokenizers.prepareTensorForDecode(tensor)` ⇒ Array Helper function to convert a tensor to a list before decoding. **Kind**: static method of [tokenizers](#module_tokenizers) **Returns**: Array - The tensor as a list. ParamTypeDescription tensorTensorThe tensor to convert. * * * ## `tokenizers._build_translation_inputs(self, raw_inputs, tokenizer_options, generate_kwargs)` ⇒ Object Helper function to build translation inputs for an `NllbTokenizer` or `M2M100Tokenizer`. **Kind**: static method of [tokenizers](#module_tokenizers) **Returns**: Object - Object to be passed to the model. ParamTypeDescription selfPreTrainedTokenizerThe tokenizer instance. raw_inputsstring | ArrayThe text to tokenize. tokenizer_optionsObjectOptions to be sent to the tokenizer generate_kwargsObjectGeneration options. * * * ## `tokenizers~PretrainedTokenizerOptions` : [PretrainedOptions](#PretrainedOptions) **Kind**: inner typedef of [tokenizers](#module_tokenizers) * * * ## `tokenizers~TextContent` : Object **Kind**: inner typedef of [tokenizers](#module_tokenizers) **Properties** NameTypeDescription type'text'The type of content (must be 'text'). textstringThe text content. * * * ## `tokenizers~ImageContent` : Object **Kind**: inner typedef of [tokenizers](#module_tokenizers) **Properties** NameTypeDescription type'image'The type of content (must be 'image'). [image]string | RawImageOptional URL or instance of the image. Note: This works for SmolVLM. Qwen2VL and Idefics3 have different implementations. * * * ## `tokenizers~MessageContent` : TextContent | ImageContent | Object Base type for message content. This is a discriminated union that can be extended with additional content types. Example: `@typedef {TextContent | ImageContent | AudioContent} MessageContent` **Kind**: inner typedef of [tokenizers](#module_tokenizers) * * * ## `tokenizers~Message` : Object **Kind**: inner typedef of [tokenizers](#module_tokenizers) **Properties** NameTypeDescription role'user' | 'assistant' | 'system' | stringThe role of the message. contentstring | ArrayThe content of the message. Can be a simple string or an array of content objects. * * * ## `tokenizers~BatchEncodingArrayItem` : any **Kind**: inner typedef of [tokenizers](#module_tokenizers) * * * ## `tokenizers~BatchEncodingItem` : any **Kind**: inner typedef of [tokenizers](#module_tokenizers) * * * ## `tokenizers~BatchEncoding` : Object **Kind**: inner typedef of [tokenizers](#module_tokenizers) **Properties** NameTypeDescription input_idsTItemList of token ids to be fed to a model. attention_maskTItemList of indices specifying which tokens should be attended to by the model. [token_type_ids]TItemList of token type ids to be fed to a model. * * * ## `tokenizers~TokenizerCallOptions` : Object **Kind**: inner typedef of [tokenizers](#module_tokenizers) **Properties** NameTypeDefaultDescription [text_pair]anyOptional second sequence to be encoded. If set, must be the same type as text. [padding]boolean | 'max_length'falseWhether to pad the input sequences. [add_special_tokens]booleantrueWhether or not to add the special tokens associated with the corresponding model. [truncation]boolean | nullWhether to truncate the input sequences. [max_length]number | nullMaximum length of the returned list and optionally padding length. [return_tensor]TReturnTensortrueWhether to return the results as Tensors or arrays. [return_token_type_ids]boolean | nullWhether to return the token type ids. * * * ## `tokenizers~PreTrainedTokenizerCallback` : function **Kind**: inner typedef of [tokenizers](#module_tokenizers) * * * ## `tokenizers~ApplyChatTemplateOptions` : Object **Kind**: inner typedef of [tokenizers](#module_tokenizers) **Properties** NameTypeDefaultDescription [chat_template]string | nullnullA Jinja template to use for this conversion. [tools]Array | nullA list of tools (callable functions) that will be accessible to the model. [documents]Array.<Record> | nullDocuments that will be accessible to the model. [add_generation_prompt]booleanfalseWhether to end the prompt with the token(s) that indicate the start of an assistant message. [tokenize]TTokenizetrueWhether to tokenize the output. If false, the output will be a string. [padding]booleanfalseWhether to pad sequences to the maximum length. Has no effect if tokenize is false. [truncation]booleanfalseWhether to truncate sequences to the maximum length. Has no effect if tokenize is false. [max_length]number | nullMaximum length (in tokens) to use for padding or truncation. Has no effect if tokenize is false. [return_tensor]TReturnTensortrueWhether to return the output as a Tensor or an Array. Has no effect if tokenize is false. [return_dict]TReturnDicttrueWhether to return a dictionary with named outputs. Has no effect if tokenize is false. [tokenizer_kwargs]Object{}Additional options to pass to the tokenizer. * * * ## `tokenizers~ApplyChatTemplateReturn` : any **Kind**: inner typedef of [tokenizers](#module_tokenizers) * * *